[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:06:47 dist-test-slave-q11v krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:06:47 dist-test-slave-q11v krb5kdc[395](info): set up 2 sockets
Jun 19 13:06:47 dist-test-slave-q11v krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 13:06:48.094750 388 mini_kdc.cc:121] Time spent starting KDC: real 2.663s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:06:48 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:06:48.274183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:48.619544 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:48.619848 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:48.619999 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:48.627341 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:48.627470 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:48.627553 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:48.627646 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:48.627733 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:48.638131 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:48.639566 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:48.640969 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:48.653777 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:48.653820 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:48.653864 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:48.654760 408 server_base.cc:1034] running on GCE node
I20260619 13:06:48.655675 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:48.657243 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:48.658519 408 hybrid_clock.cc:648] HybridClock initialized: now 1781874408658422 us; error 95 us; skew 500 ppm
I20260619 13:06:48.658897 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:48 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:48.662266 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:06:48.663955 408 webserver.cc:460] Webserver started at http://127.0.97.62:43419/ using document root <none> and password file <none>
I20260619 13:06:48.664682 408 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:48.664827 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:48.665166 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:48.668298 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "431c02257b0a41d38ed82679af07f849"
format_stamp: "Formatted at 2026-06-19 13:06:48 on dist-test-slave-q11v"
server_key: "38563116ca8a045087c78e07b7e60caf"
server_key_iv: "390934e859fa631c1b1c1f2f74a1251a"
server_key_version: "encryptionkey@0"
I20260619 13:06:48.669108 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "431c02257b0a41d38ed82679af07f849"
format_stamp: "Formatted at 2026-06-19 13:06:48 on dist-test-slave-q11v"
server_key: "38563116ca8a045087c78e07b7e60caf"
server_key_iv: "390934e859fa631c1b1c1f2f74a1251a"
server_key_version: "encryptionkey@0"
I20260619 13:06:48.674590 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 13:06:48.678648 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:48.680154 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:06:48.680361 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "431c02257b0a41d38ed82679af07f849"
format_stamp: "Formatted at 2026-06-19 13:06:48 on dist-test-slave-q11v"
server_key: "38563116ca8a045087c78e07b7e60caf"
server_key_iv: "390934e859fa631c1b1c1f2f74a1251a"
server_key_version: "encryptionkey@0"
I20260619 13:06:48.680596 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:48.709327 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:48.714190 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:48.714498 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:48.735493 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45651
I20260619 13:06:48.735553 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45651 every 8 connection(s)
I20260619 13:06:48.736896 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:06:48.740661 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:48.741789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 408
I20260619 13:06:48.742077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:06:48.743906 388 external_mini_cluster.cc:1428] Setting key 127c1b3ce0a02e7aadeda42d9dcc2685
I20260619 13:06:48.751533 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849: Bootstrap starting.
I20260619 13:06:48.754635 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:48.756067 475 log.cc:824] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:48.759639 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849: No bootstrap required, opened a new log
I20260619 13:06:48.769261 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260619 13:06:48.769793 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:48.769905 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c02257b0a41d38ed82679af07f849, State: Initialized, Role: FOLLOWER
I20260619 13:06:48.770543 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260619 13:06:48.770797 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:06:48.770951 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:06:48.771134 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:48.773370 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
Jun 19 13:06:48 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:48.774039 475 leader_election.cc:302] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431c02257b0a41d38ed82679af07f849; no voters:
I20260619 13:06:48.774736 475 leader_election.cc:288] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:06:48.774901 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:48.776432 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [term 1 LEADER]: Becoming Leader. State: Replica: 431c02257b0a41d38ed82679af07f849, State: Running, Role: LEADER
I20260619 13:06:48.777148 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260619 13:06:48.777828 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:06:48.780452 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431c02257b0a41d38ed82679af07f849. Latest consensus state: current_term: 1 leader_uuid: "431c02257b0a41d38ed82679af07f849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260619 13:06:48.780932 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:48.780902 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431c02257b0a41d38ed82679af07f849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431c02257b0a41d38ed82679af07f849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260619 13:06:48.781236 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:48.791656 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:06:48.793568 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:48.748185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.0.97.62:45651)
0619 13:06:48.749193 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:48.749214 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:48.750578 (+ 1364us) server_negotiation.cc:407] Connection header received
0619 13:06:48.759897 (+ 9319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:48.759930 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:48.760484 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:48.760972 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:48.764005 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:48.766164 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:48.767909 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:48.769028 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:48.769218 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:48.775909 (+ 6691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:48.775920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:48.775939 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:48.775990 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:48.781748 (+ 5758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:48.783290 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:48.783304 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:48.783315 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:48.783590 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:48.784382 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:48.784390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:48.784396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:48.785445 (+ 1049us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:48.790838 (+ 5393us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:48.791269 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 13:06:48.791886 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":425,"threads_started":1}
I20260619 13:06:48.797326 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:06:48.806910 487 catalog_manager.cc:1347] Generated new cluster ID: 23c0c8d3da314631a6c0c54ff6b7451c
I20260619 13:06:48.807155 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:06:48.833240 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:06:48.834666 487 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:06:48.846297 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431c02257b0a41d38ed82679af07f849: Generated new TSK 0
I20260619 13:06:48.847429 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:06:48.901937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:49.250589 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:49.250913 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:49.251080 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:49.258600 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:49.258746 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:49.258934 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:06:49.269210 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:49.270673 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:49.272042 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:49.286633 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:49.286790 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:49.286720 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:49.287832 501 server_base.cc:1034] running on GCE node
I20260619 13:06:49.288872 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:49.290112 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:49.291359 501 hybrid_clock.cc:648] HybridClock initialized: now 1781874409291303 us; error 49 us; skew 500 ppm
I20260619 13:06:49.291738 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:49 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:49.295286 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:06:49.296876 501 webserver.cc:460] Webserver started at http://127.0.97.1:33307/ using document root <none> and password file <none>
I20260619 13:06:49.297567 501 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:49.297739 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:49.298017 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:49.300976 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "779e2976c1ed4d45ab41cf8852d2a8cf"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "e1e5e085aad3868a9e5c4f2fb4ff1f4b"
server_key_iv: "22c98ff3c69611670a62242b6d0f2d82"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.301824 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "779e2976c1ed4d45ab41cf8852d2a8cf"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "e1e5e085aad3868a9e5c4f2fb4ff1f4b"
server_key_iv: "22c98ff3c69611670a62242b6d0f2d82"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.307176 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:06:49.310889 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:49.312359 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:06:49.312501 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "779e2976c1ed4d45ab41cf8852d2a8cf"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "e1e5e085aad3868a9e5c4f2fb4ff1f4b"
server_key_iv: "22c98ff3c69611670a62242b6d0f2d82"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.312673 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:49.344094 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:49.348610 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:49.348851 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:49.349920 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:49.351187 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:49.351267 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:49.351356 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:49.351406 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:49.386891 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41005
I20260619 13:06:49.386932 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41005 every 8 connection(s)
I20260619 13:06:49.388285 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:06:49.391188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 501
I20260619 13:06:49.391574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:06:49.392524 388 external_mini_cluster.cc:1428] Setting key cbcfcaaf80f9aca0b47665059ed53561
Jun 19 13:06:49 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:06:49.415195 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:49.391498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43477 (local address 127.0.97.62:45651)
0619 13:06:49.392038 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:49.392046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:49.394166 (+ 2120us) server_negotiation.cc:407] Connection header received
0619 13:06:49.397021 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:49.397028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:49.397130 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:49.397331 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:49.399596 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.401104 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.402891 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.403631 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.403779 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:49.407609 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:49.407617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:49.407624 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:49.407670 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:49.411155 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:49.412297 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:49.412301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:49.412304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:49.412517 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:49.413058 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:49.413064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:49.413068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:49.413394 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:49.413577 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:49.414452 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 13:06:49.414730 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":239,"threads_started":1}
I20260619 13:06:49.416599 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:49.392100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:43477)
0619 13:06:49.393957 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:49.394021 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:49.396653 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:49.397724 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:49.397737 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:49.398383 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:49.399347 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:49.399379 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.401342 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.401346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:49.402725 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:49.402738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.403882 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.403887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:49.404094 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:49.404903 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:49.404928 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:49.407324 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:49.411399 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:49.411420 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:49.411437 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:49.412088 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:49.412662 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:49.412669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:49.412673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:49.412913 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:49.413549 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:49.413559 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:49.414002 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 13:06:49.414325 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 13:06:49.414945 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"thread_start_us":1193,"threads_started":1}
I20260619 13:06:49.418560 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260619 13:06:49.418910 629 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:49.419664 629 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260619 13:06:49.421973 440 ts_manager.cc:194] Registered new tserver with Master: 779e2976c1ed4d45ab41cf8852d2a8cf (127.0.97.1:41005)
I20260619 13:06:49.423458 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:06:49.437573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:49.780246 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:49.780543 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:49.780645 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:49.788335 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:49.788455 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:49.788571 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:06:49.804744 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:49.806270 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:49.807542 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:49.821166 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:49.821218 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:49.821260 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:49.821965 635 server_base.cc:1034] running on GCE node
I20260619 13:06:49.822726 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:49.823701 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:49.824927 635 hybrid_clock.cc:648] HybridClock initialized: now 1781874409824869 us; error 58 us; skew 500 ppm
I20260619 13:06:49.825243 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:49 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:49.828492 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:06:49.830013 635 webserver.cc:460] Webserver started at http://127.0.97.2:34155/ using document root <none> and password file <none>
I20260619 13:06:49.830693 635 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:49.830788 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:49.831037 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:49.834044 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "23fce4b8f6f9499592ad63c45ee4955e"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "b540f4ed76d8d678bf22ba51b424214f"
server_key_iv: "90a1ac790cf63d625c05d173f32366ff"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.834789 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "23fce4b8f6f9499592ad63c45ee4955e"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "b540f4ed76d8d678bf22ba51b424214f"
server_key_iv: "90a1ac790cf63d625c05d173f32366ff"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.840590 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:06:49.844219 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:49.845789 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:49.845973 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "23fce4b8f6f9499592ad63c45ee4955e"
format_stamp: "Formatted at 2026-06-19 13:06:49 on dist-test-slave-q11v"
server_key: "b540f4ed76d8d678bf22ba51b424214f"
server_key_iv: "90a1ac790cf63d625c05d173f32366ff"
server_key_version: "encryptionkey@0"
I20260619 13:06:49.846182 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:49.864240 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:49.868989 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:49.869341 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:49.870436 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:49.871829 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:49.871937 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:49.872498 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:49.872581 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 13:06:49.908631 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43957
I20260619 13:06:49.908650 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43957 every 8 connection(s)
I20260619 13:06:49.910068 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:06:49.917392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 635
I20260619 13:06:49.917587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:06:49.918058 388 external_mini_cluster.cc:1428] Setting key 9f6adec75cf2fc529508907b9e0e0b65
Jun 19 13:06:49 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:49.935065 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:49.912740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56419 (local address 127.0.97.62:45651)
0619 13:06:49.912921 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:49.912925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:49.914257 (+ 1332us) server_negotiation.cc:407] Connection header received
0619 13:06:49.916219 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:49.916224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:49.916320 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:49.916431 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:49.918384 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.919333 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.922106 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.922556 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.922655 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:49.927528 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:49.927538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:49.927546 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:49.927595 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:49.931335 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:49.932232 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:49.932238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:49.932243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:49.932412 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:49.932951 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:49.932958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:49.932965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:49.933397 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:49.933572 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:49.934390 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 13:06:49.934636 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:49.935844 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:49.913296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:56419)
0619 13:06:49.914089 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:49.914133 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:49.915816 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:49.916705 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:49.916714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:49.917485 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:49.918221 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:49.918240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.919509 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.919513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:49.921956 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:49.921976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:49.922794 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:49.922804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:49.923137 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:49.923892 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:49.923931 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:49.927246 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:49.931502 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:49.931511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:49.931535 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:49.932061 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:49.932574 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:49.932578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:49.932581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:49.932782 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:49.933557 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:49.933564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:49.933991 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 13:06:49.934267 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 13:06:49.934673 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:06:49.937579 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260619 13:06:49.937970 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:06:49.938851 763 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260619 13:06:49.940653 440 ts_manager.cc:194] Registered new tserver with Master: 23fce4b8f6f9499592ad63c45ee4955e (127.0.97.2:43957)
I20260619 13:06:49.941992 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:06:49.961581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:50.307195 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:50.307541 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:50.307719 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:50.314994 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:50.315162 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:50.315340 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:06:50.325443 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:50.326756 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:50.328110 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:50.341679 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:50.341631 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:50.341718 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:50.342864 768 server_base.cc:1034] running on GCE node
I20260619 13:06:50.343494 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:50.344414 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:50.345682 768 hybrid_clock.cc:648] HybridClock initialized: now 1781874410345637 us; error 46 us; skew 500 ppm
I20260619 13:06:50.346033 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:50.349119 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:06:50.350852 768 webserver.cc:460] Webserver started at http://127.0.97.3:34727/ using document root <none> and password file <none>
I20260619 13:06:50.351615 768 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:50.351718 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:50.351982 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:50.354905 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "4e8ffecfbb704e01b95f95b165248385"
format_stamp: "Formatted at 2026-06-19 13:06:50 on dist-test-slave-q11v"
server_key: "54730fcdab3a224b9a1cfee872800cc6"
server_key_iv: "e27c4a7073c0eba8689d6ea876824033"
server_key_version: "encryptionkey@0"
I20260619 13:06:50.356132 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4e8ffecfbb704e01b95f95b165248385"
format_stamp: "Formatted at 2026-06-19 13:06:50 on dist-test-slave-q11v"
server_key: "54730fcdab3a224b9a1cfee872800cc6"
server_key_iv: "e27c4a7073c0eba8689d6ea876824033"
server_key_version: "encryptionkey@0"
I20260619 13:06:50.361474 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:06:50.365207 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:50.366700 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:50.366892 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "4e8ffecfbb704e01b95f95b165248385"
format_stamp: "Formatted at 2026-06-19 13:06:50 on dist-test-slave-q11v"
server_key: "54730fcdab3a224b9a1cfee872800cc6"
server_key_iv: "e27c4a7073c0eba8689d6ea876824033"
server_key_version: "encryptionkey@0"
I20260619 13:06:50.367079 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:50.388538 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:50.393667 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:50.394022 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:50.395097 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:50.396371 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:50.396446 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:50.396616 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:50.396678 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:50.426744 629 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260619 13:06:50.433152 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39613
I20260619 13:06:50.433199 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39613 every 8 connection(s)
I20260619 13:06:50.434500 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:06:50.440465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 768
I20260619 13:06:50.440675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:06:50.441234 388 external_mini_cluster.cc:1428] Setting key 7e5925e781100861b036d4c258aa26ec
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:50.455482 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.437095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59807 (local address 127.0.97.62:45651)
0619 13:06:50.437630 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.437639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.438494 (+ 855us) server_negotiation.cc:407] Connection header received
0619 13:06:50.441409 (+ 2915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.441414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.441486 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.441583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.443558 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.444513 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.446164 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.446568 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.446706 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.450234 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.450239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.450242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.450283 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.452774 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.453470 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.453473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.453476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.453589 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.454067 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.454070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.454072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.454317 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.454427 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.455068 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.455224 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":275,"threads_started":1}
I20260619 13:06:50.456770 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.437572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:59807)
0619 13:06:50.438331 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:50.438375 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:50.441005 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:50.441927 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:50.441939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:50.442558 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:50.443373 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:50.443402 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.444671 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.444681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:50.446017 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:50.446029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.446802 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.446808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:50.447035 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.447665 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:50.447691 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:50.449953 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:50.452919 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:50.452926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:50.452938 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:50.453362 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:50.453763 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:50.453767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:50.453769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:50.453962 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:50.454423 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:50.454430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:50.454734 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 13:06:50.454972 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 13:06:50.455413 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":261,"threads_started":1}
I20260619 13:06:50.458185 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260619 13:06:50.458439 896 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:50.459277 896 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260619 13:06:50.460604 440 ts_manager.cc:194] Registered new tserver with Master: 4e8ffecfbb704e01b95f95b165248385 (127.0.97.3:39613)
I20260619 13:06:50.461423 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59807
I20260619 13:06:50.469813 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:50.517191 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.499881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.97.1:41005)
0619 13:06:50.500551 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.500562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.500674 (+ 112us) server_negotiation.cc:407] Connection header received
0619 13:06:50.500928 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.500936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.501173 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.501346 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.502999 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.504066 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.505636 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.506251 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.506407 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.510545 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.510558 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.510575 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.510614 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.513685 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.514803 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.514811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.514813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.514939 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.515335 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.515341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.515344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.515753 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.516059 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.516422 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.516767 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":290,"threads_started":1}
W20260619 13:06:50.518591 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52284
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:50.563843 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.549805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.0.97.1:41005)
0619 13:06:50.550032 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.550038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.550319 (+ 281us) server_negotiation.cc:407] Connection header received
0619 13:06:50.550597 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.550601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.550695 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.550789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.552068 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.552778 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.554183 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.554627 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.554749 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.558503 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.558509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.558512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.558549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.561240 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.562035 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.562040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.562043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.562233 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.562641 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.562645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.562647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.562926 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.563105 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.563417 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.563587 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:06:50.565459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 501
I20260619 13:06:50.579473 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.566539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.97.1:41005)
0619 13:06:50.566699 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.566703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.567059 (+ 356us) server_negotiation.cc:407] Connection header received
0619 13:06:50.567176 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.567179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.567261 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.567338 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.568903 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.569582 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.571097 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.571523 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.571667 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.574090 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.574102 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.574105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.574139 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.576761 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.577432 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.577436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.577439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.577565 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.578080 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.578084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.578087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.578384 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.578566 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.578934 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.579163 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:06:50.580245 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:50.663971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 501
I20260619 13:06:50.680634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 635
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:06:50.697752 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.681660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.97.2:43957)
0619 13:06:50.682173 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.682181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.682208 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:06:50.682274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.682281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.682502 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.682658 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.684008 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.685166 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.686750 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.687281 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.687386 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.691597 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.691609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.691624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.691659 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.694852 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.695538 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.695544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.695546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.695663 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.696087 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.696090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.696092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.696429 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.696661 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.697103 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.697301 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260619 13:06:50.699031 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:50.788820 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 635
I20260619 13:06:50.812219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 768
Jun 19 13:06:50 dist-test-slave-q11v krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:06:50.828426 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:50.813257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.0.97.3:39613)
0619 13:06:50.813930 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:50.813938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:50.813967 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:06:50.814026 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:50.814032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:50.814264 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:50.814431 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:50.815707 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.816746 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.818200 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:50.818775 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:50.818919 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:50.823106 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:50.823117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:50.823131 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:50.823169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:50.826080 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.826674 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.826681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.826683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.826812 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:50.827146 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:50.827149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:50.827151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:50.827474 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:50.827689 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:50.827949 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 13:06:50.828144 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":354,"threads_started":1}
I20260619 13:06:50.829838 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:50.912171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 768
I20260619 13:06:50.928462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 408
I20260619 13:06:50.929770 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:51.031431 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 408
2026-06-19T13:06:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5652 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:06:51 dist-test-slave-q11v krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:06:51 dist-test-slave-q11v krb5kdc[918](info): set up 2 sockets
Jun 19 13:06:51 dist-test-slave-q11v krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260619 13:06:52.125506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:06:52 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:06:52.309202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:52.628721 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:52.629033 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:52.629122 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:52.636250 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:52.636363 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:52.636406 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:52.636456 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:52.636498 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:52.646297 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:52.647603 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:52.648931 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:52.661074 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:52.661033 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:52.661095 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:52.661942 931 server_base.cc:1034] running on GCE node
I20260619 13:06:52.662827 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:52.664315 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:52.665556 931 hybrid_clock.cc:648] HybridClock initialized: now 1781874412665493 us; error 80 us; skew 500 ppm
I20260619 13:06:52.665966 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:52 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:52.675168 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:06:52.676893 931 webserver.cc:460] Webserver started at http://127.0.97.62:45303/ using document root <none> and password file <none>
I20260619 13:06:52.677713 931 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:52.677863 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:52.678203 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:52.681299 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "e1790042fc084152b5bc878f47167731"
format_stamp: "Formatted at 2026-06-19 13:06:52 on dist-test-slave-q11v"
server_key: "b4c353a35e30c3becc68056de575e388"
server_key_iv: "4260bf6389a15a30ae88a1ba6b08e71f"
server_key_version: "encryptionkey@0"
I20260619 13:06:52.682183 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1790042fc084152b5bc878f47167731"
format_stamp: "Formatted at 2026-06-19 13:06:52 on dist-test-slave-q11v"
server_key: "b4c353a35e30c3becc68056de575e388"
server_key_iv: "4260bf6389a15a30ae88a1ba6b08e71f"
server_key_version: "encryptionkey@0"
I20260619 13:06:52.687613 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:06:52.691215 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:52.692724 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:06:52.692914 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "e1790042fc084152b5bc878f47167731"
format_stamp: "Formatted at 2026-06-19 13:06:52 on dist-test-slave-q11v"
server_key: "b4c353a35e30c3becc68056de575e388"
server_key_iv: "4260bf6389a15a30ae88a1ba6b08e71f"
server_key_version: "encryptionkey@0"
I20260619 13:06:52.693130 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:52.707041 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:52.711683 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:52.711990 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:52.730777 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41437
I20260619 13:06:52.730796 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41437 every 8 connection(s)
I20260619 13:06:52.732251 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:06:52.736239 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:52.736418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 931
I20260619 13:06:52.736649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:06:52.737463 388 external_mini_cluster.cc:1428] Setting key 9ee97989741ae994e6422f47cf5fc9a2
I20260619 13:06:52.749007 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731: Bootstrap starting.
Jun 19 13:06:52 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:52.752538 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:52.753710 998 log.cc:824] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:52.756718 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731: No bootstrap required, opened a new log
I20260619 13:06:52.760668 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:52.739701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.0.97.62:41437)
0619 13:06:52.740424 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:52.740436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:52.740475 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:06:52.742690 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:52.742716 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:52.743134 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:52.743555 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:52.745131 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:52.746552 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:52.747851 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:52.748546 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:52.748768 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:52.753081 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:52.753089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:52.753106 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:52.753148 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:52.756280 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:52.757063 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:52.757071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:52.757078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:52.757254 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:52.757725 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:52.757733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:52.757737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:52.758302 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:52.758634 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:52.759122 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 13:06:52.759568 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":315,"threads_started":1}
I20260619 13:06:52.764997 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } }
I20260619 13:06:52.765375 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:52.765513 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1790042fc084152b5bc878f47167731, State: Initialized, Role: FOLLOWER
I20260619 13:06:52.766201 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } }
I20260619 13:06:52.766436 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:06:52.766568 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:06:52.766741 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:52.769064 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } }
I20260619 13:06:52.769721 998 leader_election.cc:302] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1790042fc084152b5bc878f47167731; no voters:
I20260619 13:06:52.770476 998 leader_election.cc:288] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:06:52.770604 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:52.771361 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [term 1 LEADER]: Becoming Leader. State: Replica: e1790042fc084152b5bc878f47167731, State: Running, Role: LEADER
I20260619 13:06:52.771919 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } }
I20260619 13:06:52.772625 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:06:52.774893 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1790042fc084152b5bc878f47167731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } } }
I20260619 13:06:52.775431 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:52.775332 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1790042fc084152b5bc878f47167731. Latest consensus state: current_term: 1 leader_uuid: "e1790042fc084152b5bc878f47167731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1790042fc084152b5bc878f47167731" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41437 } } }
I20260619 13:06:52.775636 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:52.777827 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:06:52.784214 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:06:52.793711 1010 catalog_manager.cc:1347] Generated new cluster ID: 07ba1c58f47a4e9a9213b2072209be0d
I20260619 13:06:52.793838 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:06:52.821755 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:06:52.823688 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:06:52.837916 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1790042fc084152b5bc878f47167731: Generated new TSK 0
I20260619 13:06:52.839020 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:06:52.894117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:53.223819 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:53.224299 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:53.224476 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:53.231874 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:53.232031 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:53.232208 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:06:53.242233 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:53.243556 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:53.244930 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:53.258206 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:53.258255 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:53.258280 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:53.259424 1024 server_base.cc:1034] running on GCE node
I20260619 13:06:53.260141 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:53.261189 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:53.262418 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781874413262376 us; error 45 us; skew 500 ppm
I20260619 13:06:53.262776 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:53 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:53.266026 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:06:53.267570 1024 webserver.cc:460] Webserver started at http://127.0.97.1:40867/ using document root <none> and password file <none>
I20260619 13:06:53.268296 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:53.268442 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:53.268769 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:53.271904 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "f50da9d71dc54d08bef93ff5af221211"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "7a2eb909edfdc000b325c9e095a10cb6"
server_key_iv: "fa3ac5aa436a39f6f74f3b36284f79ce"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.272814 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f50da9d71dc54d08bef93ff5af221211"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "7a2eb909edfdc000b325c9e095a10cb6"
server_key_iv: "fa3ac5aa436a39f6f74f3b36284f79ce"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.278689 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:06:53.282534 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.284015 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:06:53.284215 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "f50da9d71dc54d08bef93ff5af221211"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "7a2eb909edfdc000b325c9e095a10cb6"
server_key_iv: "fa3ac5aa436a39f6f74f3b36284f79ce"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.284428 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:53.306439 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:53.311012 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:53.311319 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:53.312354 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:53.313776 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:53.313880 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.314033 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:53.314106 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.347509 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33315
I20260619 13:06:53.347597 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33315 every 8 connection(s)
I20260619 13:06:53.348810 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:06:53.353514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1024
I20260619 13:06:53.353765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:06:53.354533 388 external_mini_cluster.cc:1428] Setting key 50049323c7d7ea2a990fe3cabf8b269c
Jun 19 13:06:53 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:06:53.372304 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:53.351396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50429 (local address 127.0.97.62:41437)
0619 13:06:53.351866 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:53.351873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:53.352844 (+ 971us) server_negotiation.cc:407] Connection header received
0619 13:06:53.355504 (+ 2660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:53.355508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:53.355597 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:53.355718 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:53.357733 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.358463 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.361053 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.361508 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.361696 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:53.365497 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:53.365503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:53.365506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:53.365540 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:53.368315 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:53.369332 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:53.369339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:53.369345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:53.369509 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:53.370092 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:53.370097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:53.370101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:53.370480 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:53.370638 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:53.371682 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0619 13:06:53.371980 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20260619 13:06:53.373924 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:53.351976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41437 (local address 127.0.97.1:50429)
0619 13:06:53.352681 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:53.352723 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:53.355178 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:53.355935 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:53.355947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:53.356677 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:53.357530 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:53.357562 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.358631 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.358639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:53.360896 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:53.360913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.361813 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.361820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:53.362098 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:53.362813 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:53.362835 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:53.365240 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:53.368457 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:53.368470 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:53.368493 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:53.369178 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:53.369663 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:53.369670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:53.369676 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:53.369921 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:53.370633 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:53.370644 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:53.371143 (+ 499us) client_negotiation.cc:769] Sending connection context
0619 13:06:53.371492 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 13:06:53.372149 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":266,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:06:53.376060 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41437
I20260619 13:06:53.376529 1152 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:53.377702 1152 heartbeater.cc:511] Master 127.0.97.62:41437 requested a full tablet report, sending...
I20260619 13:06:53.380609 963 ts_manager.cc:194] Registered new tserver with Master: f50da9d71dc54d08bef93ff5af221211 (127.0.97.1:33315)
I20260619 13:06:53.382611 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:06:53.401149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:53.728186 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:53.728536 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:53.728711 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:53.736052 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:53.736217 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:53.736443 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:06:53.746346 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:53.747671 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:53.748937 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:53.762233 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:53.762321 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:53.762295 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:53.763140 1158 server_base.cc:1034] running on GCE node
I20260619 13:06:53.763736 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:53.764719 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:53.765923 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781874413765854 us; error 65 us; skew 500 ppm
I20260619 13:06:53.766247 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:53 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:53.769680 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:06:53.771463 1158 webserver.cc:460] Webserver started at http://127.0.97.2:41643/ using document root <none> and password file <none>
I20260619 13:06:53.772228 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:53.772416 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:53.772859 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:53.776304 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "c701215746f1442c90406024288d4934"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "d02f63229d7c48b7fc6aae1e7bc41253"
server_key_iv: "735389fdf205d0d9098e2b1b63d7c524"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.777357 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c701215746f1442c90406024288d4934"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "d02f63229d7c48b7fc6aae1e7bc41253"
server_key_iv: "735389fdf205d0d9098e2b1b63d7c524"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.783255 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:06:53.787032 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.788659 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:06:53.788865 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "c701215746f1442c90406024288d4934"
format_stamp: "Formatted at 2026-06-19 13:06:53 on dist-test-slave-q11v"
server_key: "d02f63229d7c48b7fc6aae1e7bc41253"
server_key_iv: "735389fdf205d0d9098e2b1b63d7c524"
server_key_version: "encryptionkey@0"
I20260619 13:06:53.789116 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:53.804239 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:53.808961 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:53.809296 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:53.810356 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:53.811780 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:53.811903 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.812063 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:53.812147 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:53.848793 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41077
I20260619 13:06:53.848829 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41077 every 8 connection(s)
I20260619 13:06:53.850356 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:06:53.858800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1158
I20260619 13:06:53.859017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:06:53.859588 388 external_mini_cluster.cc:1428] Setting key fa054908b756629dd640843451ee3879
Jun 19 13:06:53 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:53.872412 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:53.853083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55757 (local address 127.0.97.62:41437)
0619 13:06:53.853302 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:53.853309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:53.854641 (+ 1332us) server_negotiation.cc:407] Connection header received
0619 13:06:53.856399 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:53.856403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:53.856479 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:53.856576 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:53.858382 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.859220 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.861348 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.861973 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.862077 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:53.866518 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:53.866523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:53.866527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:53.866561 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:53.869191 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:53.870085 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:53.870090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:53.870092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:53.870216 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:53.870653 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:53.870662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:53.870664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:53.870943 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:53.871094 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:53.871842 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 13:06:53.872071 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:06:53.873559 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:53.853736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41437 (local address 127.0.97.2:55757)
0619 13:06:53.854463 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:53.854502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:53.856100 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:53.856894 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:53.856903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:53.857476 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:53.858196 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:53.858217 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.859399 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.859406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:53.861189 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:53.861204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:53.862213 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:53.862221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:53.862542 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:53.863338 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:53.863376 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:53.866257 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:53.869399 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:53.869408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:53.869427 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:53.869948 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:53.870328 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:53.870333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:53.870337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:53.870530 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:53.871069 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:53.871078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:53.871465 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 13:06:53.871734 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 13:06:53.872289 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":293,"threads_started":1}
I20260619 13:06:53.875267 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41437
I20260619 13:06:53.875613 1286 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:06:53.876456 1286 heartbeater.cc:511] Master 127.0.97.62:41437 requested a full tablet report, sending...
I20260619 13:06:53.878177 963 ts_manager.cc:194] Registered new tserver with Master: c701215746f1442c90406024288d4934 (127.0.97.2:41077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:06:53.879066 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:06:53.899147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:54.220268 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:54.220582 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:54.220681 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:54.227972 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:54.228083 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:54.228201 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:06:54.237910 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:54.239153 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:54.240422 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:54.253552 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:54.253631 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:54.253552 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:54.254312 1291 server_base.cc:1034] running on GCE node
I20260619 13:06:54.254976 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:54.255944 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:54.257200 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781874414257145 us; error 44 us; skew 500 ppm
I20260619 13:06:54.257688 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:54.261415 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:06:54.263126 1291 webserver.cc:460] Webserver started at http://127.0.97.3:44787/ using document root <none> and password file <none>
I20260619 13:06:54.263885 1291 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:54.264047 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:54.264393 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:54.267633 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "059ddfab58384bf18826bac01d79ed5a"
format_stamp: "Formatted at 2026-06-19 13:06:54 on dist-test-slave-q11v"
server_key: "9164b8237e716ba10d8bfecc7938fbff"
server_key_iv: "57ce4fcd69d6043e5c205bcc1ddcf85a"
server_key_version: "encryptionkey@0"
I20260619 13:06:54.268563 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "059ddfab58384bf18826bac01d79ed5a"
format_stamp: "Formatted at 2026-06-19 13:06:54 on dist-test-slave-q11v"
server_key: "9164b8237e716ba10d8bfecc7938fbff"
server_key_iv: "57ce4fcd69d6043e5c205bcc1ddcf85a"
server_key_version: "encryptionkey@0"
I20260619 13:06:54.274629 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 13:06:54.278510 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:54.279920 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:06:54.280117 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "059ddfab58384bf18826bac01d79ed5a"
format_stamp: "Formatted at 2026-06-19 13:06:54 on dist-test-slave-q11v"
server_key: "9164b8237e716ba10d8bfecc7938fbff"
server_key_iv: "57ce4fcd69d6043e5c205bcc1ddcf85a"
server_key_version: "encryptionkey@0"
I20260619 13:06:54.280341 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:54.299573 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:54.304167 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:54.304461 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:54.305429 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:06:54.306835 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:54.306950 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:54.307096 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:54.307168 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:54.339357 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45049
I20260619 13:06:54.339406 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45049 every 8 connection(s)
I20260619 13:06:54.340667 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:06:54.346354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1291
I20260619 13:06:54.346532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:06:54.346948 388 external_mini_cluster.cc:1428] Setting key bb4e9209545b418b27a1d4e65312d1d5
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:54.363487 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.343356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51017 (local address 127.0.97.62:41437)
0619 13:06:54.343567 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.343571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.344905 (+ 1334us) server_negotiation.cc:407] Connection header received
0619 13:06:54.347148 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.347155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.347252 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.347335 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.349700 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.350465 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.352887 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.353324 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.353489 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.357093 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.357098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.357101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.357132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.359824 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.360816 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.360821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.360825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.360992 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.361490 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.361495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.361498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.361885 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.362033 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.363049 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.363225 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 13:06:54.364518 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.343885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41437 (local address 127.0.97.3:51017)
0619 13:06:54.344682 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.344728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.346770 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.347900 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.347914 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.348697 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:54.349502 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.349531 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.350635 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.350642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.352681 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.352696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.353586 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.353640 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.353870 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.354557 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:54.354584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:54.356878 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:54.360007 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:54.360018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:54.360037 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:54.360653 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:54.361136 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:54.361141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:54.361144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:54.361346 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:54.362076 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:54.362085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:54.362588 (+ 503us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.362900 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.363320 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":341,"threads_started":1}
I20260619 13:06:54.366005 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41437
I20260619 13:06:54.366325 1419 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:54.367527 1419 heartbeater.cc:511] Master 127.0.97.62:41437 requested a full tablet report, sending...
I20260619 13:06:54.369027 963 ts_manager.cc:194] Registered new tserver with Master: 059ddfab58384bf18826bac01d79ed5a (127.0.97.3:45049)
I20260619 13:06:54.369889 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51017
I20260619 13:06:54.375336 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:06:54.386735 1152 heartbeater.cc:503] Master 127.0.97.62:41437 was elected leader, sending a full tablet report...
I20260619 13:06:54.396523 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.381969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.97.62:41437)
0619 13:06:54.382195 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.382204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.382646 (+ 442us) server_negotiation.cc:407] Connection header received
0619 13:06:54.382878 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.382890 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.382958 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.383073 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.384767 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.385568 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.386953 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.387420 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.387571 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.390362 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.390367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.390371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.390404 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.393732 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.394479 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.394486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.394489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.394612 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.394974 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.394981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.394984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.395356 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.395602 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.395951 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.396191 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:06:54.403604 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:06:54.410537 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:06:54.435909 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.423933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.97.3:45049)
0619 13:06:54.425576 (+ 1643us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.425588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.425652 (+ 64us) server_negotiation.cc:407] Connection header received
0619 13:06:54.425880 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.425889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.426157 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.426391 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:54.427404 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.429230 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.432419 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.434256 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.434442 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.434705 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.435361 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.435541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":1337,"threads_started":1}
I20260619 13:06:54.437009 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.423545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45049 (local address 127.0.0.1:53598)
0619 13:06:54.425369 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.425415 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.425577 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.426589 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.426597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.426628 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:54.427203 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.427216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.429419 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.429426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.432203 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.432224 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.434825 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.434838 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.435077 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.435133 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.436434 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.436645 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":4,"thread_start_us":2309,"threads_started":1}
I20260619 13:06:54.439002 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.423686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33315 (local address 127.0.0.1:40998)
0619 13:06:54.426104 (+ 2418us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.426118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.426333 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.428545 (+ 2212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.428553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.428587 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:54.429065 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.429079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.433210 (+ 4131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.433218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.436218 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.436232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.438149 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.438156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.438345 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.438378 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.438475 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.438607 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2308,"mutex_wait_us":60,"thread_start_us":1319,"threads_started":1}
I20260619 13:06:54.439092 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.423984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.97.1:33315)
0619 13:06:54.425094 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.425106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.426233 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 13:06:54.426546 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.426554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.426796 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.426979 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:54.429303 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.432974 (+ 3671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.436407 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.437936 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.438069 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.438294 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.438552 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.438722 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":645,"threads_started":1}
I20260619 13:06:54.442025 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.426395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41077 (local address 127.0.0.1:34438)
0619 13:06:54.427104 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.427126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.427236 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.435305 (+ 8069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.435309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.435336 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:54.435747 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.435758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.437439 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.437443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.439488 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.439500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.441276 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.441283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.441447 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.441479 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.441548 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.441715 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":767,"threads_started":1}
I20260619 13:06:54.441565 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 3407558a1c4248c5b2976a4e738b7d63 (DEFAULT_TABLE table=test-table [id=d41e406cf0ec408f8d1ccccfe7f5e56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:54.442628 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.427002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.0.97.2:41077)
0619 13:06:54.434764 (+ 7762us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.434775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.434808 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:06:54.434890 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.434897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.435110 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.435335 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:54.435953 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.437268 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.439649 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.441043 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.441872 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.442063 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.442132 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.442293 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7588,"thread_start_us":7445,"threads_started":1}
I20260619 13:06:54.443186 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3407558a1c4248c5b2976a4e738b7d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:54.442797 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3407558a1c4248c5b2976a4e738b7d63 (DEFAULT_TABLE table=test-table [id=d41e406cf0ec408f8d1ccccfe7f5e56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:54.444574 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3407558a1c4248c5b2976a4e738b7d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:54.446626 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3407558a1c4248c5b2976a4e738b7d63 (DEFAULT_TABLE table=test-table [id=d41e406cf0ec408f8d1ccccfe7f5e56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:54.448069 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3407558a1c4248c5b2976a4e738b7d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:54.456457 1437 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Bootstrap starting.
I20260619 13:06:54.459966 1437 tablet_bootstrap.cc:654] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:54.461062 1437 log.cc:824] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:54.465628 1438 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Bootstrap starting.
I20260619 13:06:54.467140 1440 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Bootstrap starting.
I20260619 13:06:54.469226 1437 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: No bootstrap required, opened a new log
I20260619 13:06:54.469446 1438 tablet_bootstrap.cc:654] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:54.469524 1437 ts_tablet_manager.cc:1397] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260619 13:06:54.470518 1438 log.cc:824] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:54.470685 1440 tablet_bootstrap.cc:654] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:54.472345 1440 log.cc:824] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:54.473582 1438 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: No bootstrap required, opened a new log
I20260619 13:06:54.473874 1438 ts_tablet_manager.cc:1397] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 13:06:54.475499 1440 tablet_bootstrap.cc:492] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: No bootstrap required, opened a new log
I20260619 13:06:54.475802 1440 ts_tablet_manager.cc:1397] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:06:54.481920 1437 raft_consensus.cc:348] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.482079 1438 raft_consensus.cc:348] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.482417 1437 raft_consensus.cc:374] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:54.482434 1438 raft_consensus.cc:374] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:54.482522 1438 raft_consensus.cc:729] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ddfab58384bf18826bac01d79ed5a, State: Initialized, Role: FOLLOWER
I20260619 13:06:54.482558 1437 raft_consensus.cc:729] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50da9d71dc54d08bef93ff5af221211, State: Initialized, Role: FOLLOWER
I20260619 13:06:54.483162 1438 consensus_queue.cc:260] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.483243 1437 consensus_queue.cc:260] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.484325 1419 heartbeater.cc:503] Master 127.0.97.62:41437 was elected leader, sending a full tablet report...
I20260619 13:06:54.486402 1438 ts_tablet_manager.cc:1428] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260619 13:06:54.487493 1440 raft_consensus.cc:348] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.488158 1440 raft_consensus.cc:374] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:54.488301 1440 raft_consensus.cc:729] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c701215746f1442c90406024288d4934, State: Initialized, Role: FOLLOWER
I20260619 13:06:54.488970 1440 consensus_queue.cc:260] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.493924 1437 ts_tablet_manager.cc:1428] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Time spent starting tablet: real 0.024s user 0.011s sys 0.009s
I20260619 13:06:54.494763 1440 ts_tablet_manager.cc:1428] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260619 13:06:54.495178 1286 heartbeater.cc:503] Master 127.0.97.62:41437 was elected leader, sending a full tablet report...
W20260619 13:06:54.592567 1420 tablet.cc:2369] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:06:54.601155 1153 tablet.cc:2369] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:06:54.602291 1287 tablet.cc:2369] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:06:54.608546 1445 raft_consensus.cc:482] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:06:54.608845 1445 raft_consensus.cc:504] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.610576 1445 leader_election.cc:288] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059ddfab58384bf18826bac01d79ed5a (127.0.97.3:45049), f50da9d71dc54d08bef93ff5af221211 (127.0.97.1:33315)
I20260619 13:06:54.618244 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.611572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34235 (local address 127.0.97.3:45049)
0619 13:06:54.611788 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.611797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.612122 (+ 325us) server_negotiation.cc:407] Connection header received
0619 13:06:54.612370 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.612375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.612436 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.612531 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:54.613064 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.613880 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.616197 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.617257 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.617415 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.617524 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.617850 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.617974 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:06:54.618299 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.611413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39803 (local address 127.0.97.1:33315)
0619 13:06:54.611615 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.611621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.612426 (+ 805us) server_negotiation.cc:407] Connection header received
0619 13:06:54.612625 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.612629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.612684 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.612784 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:54.613351 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.614129 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.616441 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.617378 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.617530 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.617655 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.617928 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.618048 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:06:54.618413 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.611058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45049 (local address 127.0.97.2:34235)
0619 13:06:54.611994 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.612012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.612214 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.612558 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.612563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.612587 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:54.612912 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.612921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.614012 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.614017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.616052 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.616065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.617469 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.617475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.617696 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.617720 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.617825 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.617966 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":37,"thread_start_us":908,"threads_started":1}
I20260619 13:06:54.618455 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.611058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33315 (local address 127.0.97.2:39803)
0619 13:06:54.612310 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:54.612326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:54.612487 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:54.612797 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:54.612805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:54.612829 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:54.613206 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.613215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.614267 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.614273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.616276 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:54.616289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.617619 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.617626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:54.617781 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.617804 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:06:54.617904 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 13:06:54.618027 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":71,"thread_start_us":511,"threads_started":1}
I20260619 13:06:54.619513 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3407558a1c4248c5b2976a4e738b7d63" candidate_uuid: "c701215746f1442c90406024288d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50da9d71dc54d08bef93ff5af221211" is_pre_election: true
I20260619 13:06:54.619590 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3407558a1c4248c5b2976a4e738b7d63" candidate_uuid: "c701215746f1442c90406024288d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ddfab58384bf18826bac01d79ed5a" is_pre_election: true
I20260619 13:06:54.620042 1107 raft_consensus.cc:2454] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c701215746f1442c90406024288d4934 in term 0.
I20260619 13:06:54.620132 1374 raft_consensus.cc:2454] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c701215746f1442c90406024288d4934 in term 0.
I20260619 13:06:54.620767 1176 leader_election.cc:302] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c701215746f1442c90406024288d4934, f50da9d71dc54d08bef93ff5af221211; no voters:
I20260619 13:06:54.621171 1445 raft_consensus.cc:2781] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:06:54.621313 1445 raft_consensus.cc:482] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:06:54.621416 1445 raft_consensus.cc:3037] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:54.623584 1445 raft_consensus.cc:504] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.624251 1445 leader_election.cc:288] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [CANDIDATE]: Term 1 election: Requested vote from peers 059ddfab58384bf18826bac01d79ed5a (127.0.97.3:45049), f50da9d71dc54d08bef93ff5af221211 (127.0.97.1:33315)
I20260619 13:06:54.624642 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3407558a1c4248c5b2976a4e738b7d63" candidate_uuid: "c701215746f1442c90406024288d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ddfab58384bf18826bac01d79ed5a"
I20260619 13:06:54.624742 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3407558a1c4248c5b2976a4e738b7d63" candidate_uuid: "c701215746f1442c90406024288d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50da9d71dc54d08bef93ff5af221211"
I20260619 13:06:54.624840 1374 raft_consensus.cc:3037] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:54.624976 1107 raft_consensus.cc:3037] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:54.627085 1374 raft_consensus.cc:2454] T 3407558a1c4248c5b2976a4e738b7d63 P 059ddfab58384bf18826bac01d79ed5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c701215746f1442c90406024288d4934 in term 1.
I20260619 13:06:54.627241 1107 raft_consensus.cc:2454] T 3407558a1c4248c5b2976a4e738b7d63 P f50da9d71dc54d08bef93ff5af221211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c701215746f1442c90406024288d4934 in term 1.
I20260619 13:06:54.627552 1174 leader_election.cc:302] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059ddfab58384bf18826bac01d79ed5a, c701215746f1442c90406024288d4934; no voters:
I20260619 13:06:54.628010 1445 raft_consensus.cc:2781] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:54.628455 1445 raft_consensus.cc:686] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [term 1 LEADER]: Becoming Leader. State: Replica: c701215746f1442c90406024288d4934, State: Running, Role: LEADER
I20260619 13:06:54.629128 1445 consensus_queue.cc:237] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } }
I20260619 13:06:54.633565 962 catalog_manager.cc:5495] T 3407558a1c4248c5b2976a4e738b7d63 P c701215746f1442c90406024288d4934 reported cstate change: term changed from 0 to 1, leader changed from <none> to c701215746f1442c90406024288d4934 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c701215746f1442c90406024288d4934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ddfab58384bf18826bac01d79ed5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50da9d71dc54d08bef93ff5af221211" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c701215746f1442c90406024288d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41077 } health_report { overall_health: HEALTHY } } }
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:54.681104 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.665057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.0.97.1:33315)
0619 13:06:54.665296 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.665305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.665762 (+ 457us) server_negotiation.cc:407] Connection header received
0619 13:06:54.666007 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.666014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.666092 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.666201 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.667670 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.668415 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.670202 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.670634 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.670770 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.675125 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.675142 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.675158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.675196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.678488 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.679179 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.679187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.679189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.679325 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.679734 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.679737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.679739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.680093 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.680322 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.680653 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.680825 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:54.727843 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.713095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.0.97.1:33315)
0619 13:06:54.713390 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.713399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.713745 (+ 346us) server_negotiation.cc:407] Connection header received
0619 13:06:54.714004 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.714008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.714071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.714178 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.715498 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.716231 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.717508 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.718084 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.718224 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.722602 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.722607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.722614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.722649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.725087 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.725756 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.725762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.725766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.725896 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.726317 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.726322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.726325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.726621 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.726898 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.727155 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.727376 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:54.775923 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.760595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.97.1:33315)
0619 13:06:54.760848 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.760856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.761143 (+ 287us) server_negotiation.cc:407] Connection header received
0619 13:06:54.761384 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.761389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.761454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.761570 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.762941 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.763627 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.764900 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.765327 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.765465 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.770122 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.770127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.770130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.770168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.773011 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.773908 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.773915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.773919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.774069 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.774503 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.774506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.774509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.774921 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.775123 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.775410 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.775610 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:06:54.777940 1067 tablet_service.cc:2908] Rejecting scan request for tablet 3407558a1c4248c5b2976a4e738b7d63: Uninitialized: clean time has not yet been initialized
I20260619 13:06:54.781802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1024
I20260619 13:06:54.794888 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.782796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.97.1:33315)
0619 13:06:54.782974 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.782978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.783382 (+ 404us) server_negotiation.cc:407] Connection header received
0619 13:06:54.783524 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.783527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.783595 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.783676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.785053 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.785815 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.786957 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.787341 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.787445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.789685 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.789690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.789693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.789723 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.792308 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.792930 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.792934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.792937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.793053 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.793406 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.793411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.793415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.793777 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.794068 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.794366 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.794579 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:06:54.795502 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:54.887144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1024
I20260619 13:06:54.904469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1158
Jun 19 13:06:54 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:06:54.919018 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:54.905232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.97.2:41077)
0619 13:06:54.905429 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:54.905433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:54.905483 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:06:54.905669 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:54.905672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:54.905735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:54.905825 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:54.906911 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.907653 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.908978 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:54.909375 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:54.909469 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:54.913569 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:54.913585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:54.913753 (+ 168us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:54.913791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:54.916475 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.917119 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.917126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.917129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.917309 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:54.917808 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:54.917811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:54.917813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:54.918178 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:54.918425 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:54.918612 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 13:06:54.918781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:06:54.919715 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:55.010741 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1158
I20260619 13:06:55.028187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1291
Jun 19 13:06:55 dist-test-slave-q11v krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:06:55.044257 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:55.029085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.97.3:45049)
0619 13:06:55.029267 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:55.029271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:55.029290 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:06:55.029398 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:55.029401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:55.029464 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:55.029546 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:55.030920 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:55.031643 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:55.033034 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:55.033439 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:55.033538 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:55.038016 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:55.038032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:55.038049 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:55.038086 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:55.040963 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:55.042041 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:55.042050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:55.042052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:55.042242 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:55.042724 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:55.042729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:55.042731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:55.043142 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:55.043371 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:55.043677 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:06:55.043859 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:55.044960 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:55.133085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1291
I20260619 13:06:55.149732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 931
I20260619 13:06:55.151005 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:55.258527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 931
2026-06-19T13:06:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4228 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:06:55 dist-test-slave-q11v krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:06:55 dist-test-slave-q11v krb5kdc[1467](info): set up 2 sockets
Jun 19 13:06:55 dist-test-slave-q11v krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 13:06:56.361801 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:06:56 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:06:56.547071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33445
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:56.880892 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:56.881259 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:56.881422 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:56.888970 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:56.889149 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:06:56.889249 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:56.889356 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:56.889441 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:56.889532 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:56.899780 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33445
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:56.901194 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:56.902588 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:56.914510 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:56.914546 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:56.914579 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:56.915696 1480 server_base.cc:1034] running on GCE node
I20260619 13:06:56.916500 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:56.918005 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:56.919222 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781874416919146 us; error 60 us; skew 500 ppm
I20260619 13:06:56.919552 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:56 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:56.922811 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:06:56.924292 1480 webserver.cc:460] Webserver started at http://127.0.97.62:38157/ using document root <none> and password file <none>
I20260619 13:06:56.924993 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:56.925079 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:56.925410 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:56.928406 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "700be010de2a4bb3a603dfcde2bfdee4"
format_stamp: "Formatted at 2026-06-19 13:06:56 on dist-test-slave-q11v"
server_key: "2e6995d7a027647df962b7547e91b4e4"
server_key_iv: "ea2bd14bfd1ba36ce7dd024439e27fcf"
server_key_version: "encryptionkey@0"
I20260619 13:06:56.929235 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "700be010de2a4bb3a603dfcde2bfdee4"
format_stamp: "Formatted at 2026-06-19 13:06:56 on dist-test-slave-q11v"
server_key: "2e6995d7a027647df962b7547e91b4e4"
server_key_iv: "ea2bd14bfd1ba36ce7dd024439e27fcf"
server_key_version: "encryptionkey@0"
I20260619 13:06:56.934430 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:06:56.938061 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:56.939420 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:06:56.939600 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "700be010de2a4bb3a603dfcde2bfdee4"
format_stamp: "Formatted at 2026-06-19 13:06:56 on dist-test-slave-q11v"
server_key: "2e6995d7a027647df962b7547e91b4e4"
server_key_iv: "ea2bd14bfd1ba36ce7dd024439e27fcf"
server_key_version: "encryptionkey@0"
I20260619 13:06:56.939831 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:56.968484 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:56.972981 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:56.973304 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:56.995675 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33445
I20260619 13:06:56.995718 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33445 every 8 connection(s)
I20260619 13:06:56.997009 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:06:57.000694 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:57.005896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1480
I20260619 13:06:57.006109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:06:57.006901 388 external_mini_cluster.cc:1428] Setting key 0443bffd8a0d4e57d3489d7e54bb9ece
I20260619 13:06:57.012204 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Bootstrap starting.
I20260619 13:06:57.015369 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:57.016407 1557 log.cc:824] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:57.019701 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: No bootstrap required, opened a new log
Jun 19 13:06:57 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:57.030149 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } }
I20260619 13:06:57.030612 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:57.030720 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700be010de2a4bb3a603dfcde2bfdee4, State: Initialized, Role: FOLLOWER
I20260619 13:06:57.031414 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } }
I20260619 13:06:57.031695 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:06:57.031831 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:06:57.031908 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:57.009176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.97.62:33445)
0619 13:06:57.010238 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:57.010254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:57.010301 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:06:57.012019 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:57.012041 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:57.012539 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:57.012959 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:57.014473 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.015851 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.017294 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.018303 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.018490 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:57.022765 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:57.022776 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:57.022800 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:57.022850 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:57.027101 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.027879 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.027889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.027898 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.028126 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.028528 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.028533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.028536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.029291 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:57.029556 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:57.030014 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 13:06:57.030449 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":442,"threads_started":1}
I20260619 13:06:57.032033 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:57.034816 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } }
I20260619 13:06:57.035405 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700be010de2a4bb3a603dfcde2bfdee4; no voters:
I20260619 13:06:57.035946 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:06:57.036149 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:57.036964 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [term 1 LEADER]: Becoming Leader. State: Replica: 700be010de2a4bb3a603dfcde2bfdee4, State: Running, Role: LEADER
I20260619 13:06:57.037773 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } }
I20260619 13:06:57.038146 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:06:57.042667 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700be010de2a4bb3a603dfcde2bfdee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } } }
I20260619 13:06:57.043221 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700be010de2a4bb3a603dfcde2bfdee4. Latest consensus state: current_term: 1 leader_uuid: "700be010de2a4bb3a603dfcde2bfdee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700be010de2a4bb3a603dfcde2bfdee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33445 } } }
I20260619 13:06:57.043619 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:57.044032 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:57.046681 1573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:06:57.051599 1573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:06:57.059836 1573 catalog_manager.cc:1347] Generated new cluster ID: b9ebcf6d089548b9a73c59aa33d1f263
I20260619 13:06:57.060060 1573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:06:57.083927 1573 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:06:57.086069 1573 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:06:57.101210 1573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Generated new TSK 0
I20260619 13:06:57.102449 1573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:06:57.156950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33445
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:57.503865 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:57.504155 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:57.504328 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:57.512019 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:57.512135 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:57.512280 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:06:57.512408 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:06:57.522488 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:57.523849 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:57.525254 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:57.539783 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:57.540004 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:57.540376 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:57.541119 1583 server_base.cc:1034] running on GCE node
I20260619 13:06:57.541975 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:57.543068 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:57.544319 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781874417544224 us; error 63 us; skew 500 ppm
I20260619 13:06:57.544711 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:57 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:57.548115 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:06:57.549767 1583 webserver.cc:460] Webserver started at http://127.0.97.1:43543/ using document root <none> and password file <none>
I20260619 13:06:57.550478 1583 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:57.550565 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:57.550891 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:57.553969 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "519a6598394b405ba55d82b956e20188"
format_stamp: "Formatted at 2026-06-19 13:06:57 on dist-test-slave-q11v"
server_key: "934918af909fe46109b64f7f5a17a16d"
server_key_iv: "96a1f781b2d2e28420e9ab2f05df155f"
server_key_version: "encryptionkey@0"
I20260619 13:06:57.554795 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "519a6598394b405ba55d82b956e20188"
format_stamp: "Formatted at 2026-06-19 13:06:57 on dist-test-slave-q11v"
server_key: "934918af909fe46109b64f7f5a17a16d"
server_key_iv: "96a1f781b2d2e28420e9ab2f05df155f"
server_key_version: "encryptionkey@0"
I20260619 13:06:57.560425 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:06:57.564348 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:57.565910 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:06:57.566099 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "519a6598394b405ba55d82b956e20188"
format_stamp: "Formatted at 2026-06-19 13:06:57 on dist-test-slave-q11v"
server_key: "934918af909fe46109b64f7f5a17a16d"
server_key_iv: "96a1f781b2d2e28420e9ab2f05df155f"
server_key_version: "encryptionkey@0"
I20260619 13:06:57.566350 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:57.580843 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:57.585319 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:57.585652 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:57.588286 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:57.588394 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:57.588542 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:57.588617 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:06:57 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:57.614374 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:57.589070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47893 (local address 127.0.97.62:33445)
0619 13:06:57.589505 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:57.589511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:57.590861 (+ 1350us) server_negotiation.cc:407] Connection header received
0619 13:06:57.593345 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:57.593351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:57.593445 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:57.593563 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:57.596427 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.597339 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.600661 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.601509 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.601697 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:57.605626 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:57.605637 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:57.605644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:57.605694 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:57.609124 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.610132 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.610137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.610139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.610314 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.611014 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.611020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.611024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.611582 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:57.611730 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:57.613807 (+ 2077us) server_negotiation.cc:299] Negotiation successful
0619 13:06:57.614040 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260619 13:06:57.615070 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:57.589776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.1:47893)
0619 13:06:57.590667 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:57.590717 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:57.592977 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:57.593959 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:57.593979 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:57.594997 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:57.596169 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:57.596210 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.597507 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.597514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:57.600426 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:57.600448 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.601786 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.601792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:57.601997 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:57.602804 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:57.602827 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:57.605366 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:57.609277 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:57.609287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:57.609305 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:57.609971 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:57.610558 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:57.610563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:57.610567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:57.610838 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:57.611803 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:57.611810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:57.612384 (+ 574us) client_negotiation.cc:769] Sending connection context
0619 13:06:57.612714 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 13:06:57.613341 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":385,"threads_started":1}
I20260619 13:06:57.634684 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35201
I20260619 13:06:57.634882 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35201 every 8 connection(s)
I20260619 13:06:57.636142 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:06:57.639186 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33445
I20260619 13:06:57.639446 1719 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:57.640178 1719 heartbeater.cc:511] Master 127.0.97.62:33445 requested a full tablet report, sending...
I20260619 13:06:57.640753 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:57.622684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58707 (local address 127.0.97.62:33445)
0619 13:06:57.622932 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:57.622941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:57.626212 (+ 3271us) server_negotiation.cc:407] Connection header received
0619 13:06:57.626415 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:57.626419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:57.626486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:57.626598 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:57.628197 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.629162 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.631955 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.632677 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.632808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:57.634920 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:57.634925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:57.634929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:57.634962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:57.637866 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.638591 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.638596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.638599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.638753 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:57.639198 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:57.639203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:57.639207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:57.639565 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:57.639713 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:57.640158 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 13:06:57.640395 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:06:57.640867 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:57.622792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.1:58707)
0619 13:06:57.626070 (+ 3278us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:57.626090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:57.626277 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:57.626651 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:57.626656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:57.627417 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:57.627986 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:57.628010 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.629308 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.629313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:57.631710 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:57.631729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:57.632893 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:57.632899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:57.633014 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:57.633762 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:57.633791 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:57.634689 (+ 898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:57.638005 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:57.638010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:57.638014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:57.638456 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:57.638873 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:57.638877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:57.638881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:57.639040 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:57.639691 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:57.639696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:57.639893 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 13:06:57.640028 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 13:06:57.640395 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3107,"thread_start_us":2970,"threads_started":1}
I20260619 13:06:57.641543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1583
I20260619 13:06:57.641772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:06:57.642611 388 external_mini_cluster.cc:1428] Setting key b9633285bab5ce4b239c6555703d8b47
I20260619 13:06:57.642763 1511 ts_manager.cc:194] Registered new tserver with Master: 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201)
I20260619 13:06:57.644774 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47893
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:06:57.681672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33445
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:58.011350 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:58.011648 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:58.011745 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:58.019217 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:58.019330 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:58.019413 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:06:58.019500 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:06:58.029419 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:58.030782 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:58.032068 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:58.045924 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:58.045856 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:58.045890 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:58.046793 1723 server_base.cc:1034] running on GCE node
I20260619 13:06:58.047533 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:58.048547 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:58.049778 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781874418049695 us; error 65 us; skew 500 ppm
I20260619 13:06:58.050097 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:58.056878 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:06:58.058636 1723 webserver.cc:460] Webserver started at http://127.0.97.2:40989/ using document root <none> and password file <none>
I20260619 13:06:58.059382 1723 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:58.059516 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:58.059846 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:58.062961 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b13765340c64d3fa69e51660a944231"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "648c37386471d297e525fc44e75fdcdc"
server_key_iv: "08189c01aefe90e4239aebd625dd5224"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.063786 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b13765340c64d3fa69e51660a944231"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "648c37386471d297e525fc44e75fdcdc"
server_key_iv: "08189c01aefe90e4239aebd625dd5224"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.069176 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 13:06:58.073071 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:58.074644 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:58.074827 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "8b13765340c64d3fa69e51660a944231"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "648c37386471d297e525fc44e75fdcdc"
server_key_iv: "08189c01aefe90e4239aebd625dd5224"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.075069 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:58.102625 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:58.106926 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:58.107225 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:58.110083 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:58.110199 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:58.110347 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:58.110422 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:58.135994 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.111435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.2:47661)
0619 13:06:58.112346 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.112393 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.114355 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.115295 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.115309 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.116308 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:58.117481 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.117523 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.118883 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.118891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.121767 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.121790 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.122678 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.122688 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.123145 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.124199 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:58.124222 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:58.127416 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:58.131262 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.131275 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.131302 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.131960 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.132349 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.132354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.132363 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.132678 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.133355 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:58.133364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:58.133954 (+ 590us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.134326 (+ 372us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.134782 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":392,"threads_started":1}
I20260619 13:06:58.136793 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.110799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47661 (local address 127.0.97.62:33445)
0619 13:06:58.111165 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.111171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.112539 (+ 1368us) server_negotiation.cc:407] Connection header received
0619 13:06:58.114662 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.114668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.114759 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.114872 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:58.117813 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.118678 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.121962 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.122402 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.122510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.127689 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:58.127694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:58.127698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:58.127731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:58.131044 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.132107 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.132114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.132118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.132241 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.132812 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.132817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.132821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.133241 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:58.133381 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.136160 (+ 2779us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.136416 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260619 13:06:58.168717 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.143338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34785 (local address 127.0.97.62:33445)
0619 13:06:58.143815 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.143822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.154207 (+ 10385us) server_negotiation.cc:407] Connection header received
0619 13:06:58.154783 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.154788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.154865 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.154968 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:58.155934 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.156682 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.158132 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.158894 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.159069 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.160954 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:58.160964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:58.160972 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:58.161031 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:58.165185 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.166109 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.166117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.166124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.166289 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.166826 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.166834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.166837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.167232 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:58.167385 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.167970 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.168267 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260619 13:06:58.168998 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.143371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.2:34785)
0619 13:06:58.154056 (+ 10685us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.154078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.154619 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.154994 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.154998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.155500 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:58.155791 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.155800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.156820 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.156825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.157949 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.157961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.159112 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.159117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.159242 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.159997 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:58.160013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:58.160755 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:58.165323 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.165327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.165330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.165977 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.166448 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.166460 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.166467 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.166677 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.167354 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:58.167360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:58.167610 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.167838 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.168275 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10517,"thread_start_us":10852,"threads_started":1}
I20260619 13:06:58.179423 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43583
I20260619 13:06:58.179468 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43583 every 8 connection(s)
I20260619 13:06:58.180718 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:06:58.182714 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33445
I20260619 13:06:58.182937 1858 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:58.183591 1858 heartbeater.cc:511] Master 127.0.97.62:33445 requested a full tablet report, sending...
I20260619 13:06:58.184973 1511 ts_manager.cc:194] Registered new tserver with Master: 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583)
I20260619 13:06:58.185876 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47661
I20260619 13:06:58.190575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1723
I20260619 13:06:58.190796 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:06:58.191205 388 external_mini_cluster.cc:1428] Setting key 4ea61d124e5bf8bdcf0fd66ecd75f6f6
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:06:58.230831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33445
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:58.561076 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:58.561374 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:58.561548 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:58.569337 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:58.569515 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:58.569696 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:06:58.569821 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:06:58.580490 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:58.581940 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:58.583321 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:58.597033 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:58.597086 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:58.597240 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:58.598307 1862 server_base.cc:1034] running on GCE node
I20260619 13:06:58.598949 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:58.599929 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:58.601161 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781874418601075 us; error 67 us; skew 500 ppm
I20260619 13:06:58.601526 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:58.604763 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:06:58.606338 1862 webserver.cc:460] Webserver started at http://127.0.97.3:37877/ using document root <none> and password file <none>
I20260619 13:06:58.607080 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:58.607208 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:58.607537 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:58.610524 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "9e2a6cfcb6634416a713ff83de17cba7"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "815fb4af02dd274769ac2ac1450f108b"
server_key_iv: "b06bac37bafb00c0056483d58b62d020"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.611301 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9e2a6cfcb6634416a713ff83de17cba7"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "815fb4af02dd274769ac2ac1450f108b"
server_key_iv: "b06bac37bafb00c0056483d58b62d020"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.616634 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:06:58.620304 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:58.621946 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:06:58.622134 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "9e2a6cfcb6634416a713ff83de17cba7"
format_stamp: "Formatted at 2026-06-19 13:06:58 on dist-test-slave-q11v"
server_key: "815fb4af02dd274769ac2ac1450f108b"
server_key_iv: "b06bac37bafb00c0056483d58b62d020"
server_key_version: "encryptionkey@0"
I20260619 13:06:58.622376 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:58.647663 1719 heartbeater.cc:503] Master 127.0.97.62:33445 was elected leader, sending a full tablet report...
I20260619 13:06:58.648274 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:58.652735 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:58.653030 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:58.655805 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:06:58.655894 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:58.656029 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:06:58.656097 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:58.682387 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.656790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33383 (local address 127.0.97.62:33445)
0619 13:06:58.657015 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.657022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.658686 (+ 1664us) server_negotiation.cc:407] Connection header received
0619 13:06:58.662076 (+ 3390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.662084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.662191 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.662334 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:58.665260 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.666344 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.670063 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.670489 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.670596 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.676158 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:58.676163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:58.676166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:58.676199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:58.679230 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.680026 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.680031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.680035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.680174 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.680589 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.680594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.680597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.680881 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:58.681030 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.681790 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.682027 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:06:58.683599 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.657437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.3:33383)
0619 13:06:58.658477 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.658533 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.661665 (+ 3132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.662695 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.662709 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.663761 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:58.664977 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.665021 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.666474 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.666479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.669860 (+ 3381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.669886 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.670782 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.670792 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.671219 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.672190 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:58.672233 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:58.675919 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:58.679372 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.679382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.679401 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.679894 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.680280 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.680286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.680290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.680477 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.681004 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:58.681012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:58.681375 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.681674 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.682260 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":439,"threads_started":1}
I20260619 13:06:58.706173 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.689748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36119 (local address 127.0.97.62:33445)
0619 13:06:58.689907 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.689911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.690583 (+ 672us) server_negotiation.cc:407] Connection header received
0619 13:06:58.690780 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.690786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.690856 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.690950 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:58.692339 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.693344 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.695475 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.696013 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.696147 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.698704 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:58.698711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:58.698716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:58.698757 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:58.702753 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.703476 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.703485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.703491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.703655 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.704153 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.704157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.704160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.704569 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:58.704716 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.705684 (+ 968us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.705932 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:06:58.706352 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.689849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.97.3:36119)
0619 13:06:58.690477 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.690500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.690660 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.690991 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.690996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.691701 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:58.692197 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.692210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.693488 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.693494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.695320 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.695331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.696409 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.696421 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.696650 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.697356 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:58.697390 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:58.698507 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:58.702925 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.702929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.702932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.703344 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.703786 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:58.703793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:58.703807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:58.703989 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:58.704764 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:58.704771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:58.705004 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.705188 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.705610 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":379,"threads_started":1}
I20260619 13:06:58.714293 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43173
I20260619 13:06:58.714315 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43173 every 8 connection(s)
I20260619 13:06:58.715505 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:06:58.717486 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33445
I20260619 13:06:58.717728 1997 heartbeater.cc:464] Registering TS with master...
I20260619 13:06:58.718362 1997 heartbeater.cc:511] Master 127.0.97.62:33445 requested a full tablet report, sending...
I20260619 13:06:58.719720 1511 ts_manager.cc:194] Registered new tserver with Master: 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3:43173)
I20260619 13:06:58.720350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 1862
I20260619 13:06:58.720521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:06:58.720598 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33383
I20260619 13:06:58.721055 388 external_mini_cluster.cc:1428] Setting key ab759e8528f70d6d438600eb6f253aa1
I20260619 13:06:58.723994 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:06:58 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:58.772116 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.757703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.0.97.62:33445)
0619 13:06:58.757888 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.757893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.758328 (+ 435us) server_negotiation.cc:407] Connection header received
0619 13:06:58.758459 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.758463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.758535 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.758634 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:58.760114 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.760927 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.762286 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.762717 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.762824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.766652 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:58.766658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:58.766662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:58.766699 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:58.769383 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.770183 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.770187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.770190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.770312 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:58.770779 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:58.770783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:58.770785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:58.771167 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:58.771361 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.771636 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.771825 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:06:58.776202 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:06:58.783310 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:06:58.819641 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.795715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.97.3:43173)
0619 13:06:58.810629 (+ 14914us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.810637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.810675 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:06:58.810768 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.810777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.811045 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.811247 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:58.812183 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.813764 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.816825 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.818326 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.818587 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.818873 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.819142 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.819307 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14719,"thread_start_us":14540,"threads_started":1}
I20260619 13:06:58.819780 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.795417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43173 (local address 127.0.0.1:38814)
0619 13:06:58.807269 (+ 11852us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.807339 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.807618 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.811538 (+ 3920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.811545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.811573 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:58.811980 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.811991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.813939 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.813945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.816649 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.816662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.818662 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.818670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.818932 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.818970 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.819092 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.819287 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11407,"mutex_wait_us":126,"thread_start_us":13191,"threads_started":1}
I20260619 13:06:58.820082 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.795488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.0.1:54992)
0619 13:06:58.808394 (+ 12906us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.808417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.808554 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.810881 (+ 2327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.810888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.810928 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:58.811403 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.811416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.814228 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.814236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.816648 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.816662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.818586 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.818593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.818843 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.819198 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.819286 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.819438 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12742,"mutex_wait_us":407,"thread_start_us":12572,"threads_started":1}
I20260619 13:06:58.820786 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.795724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.97.2:43583)
0619 13:06:58.810117 (+ 14393us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.810129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.810164 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:06:58.810270 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.810280 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.810639 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.811625 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:58.811815 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.814088 (+ 2273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.816823 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.818326 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.818946 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.819147 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.820305 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.820492 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14246,"thread_start_us":14133,"threads_started":1}
I20260619 13:06:58.823799 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 (DEFAULT_TABLE table=test-table [id=59a4b0d7c5504d87915c79799107507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:58.823807 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 (DEFAULT_TABLE table=test-table [id=59a4b0d7c5504d87915c79799107507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:58.825111 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73758a037f2a435b9a8298d5011b8ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:58.825116 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73758a037f2a435b9a8298d5011b8ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:58.825992 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.810714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35201 (local address 127.0.0.1:50862)
0619 13:06:58.817555 (+ 6841us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.817570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.817739 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.818273 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.818277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.818302 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:58.818763 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.818774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.820561 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.820566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.823258 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.823274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.825076 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.825083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.825385 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.825427 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.825527 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.825685 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6725,"thread_start_us":6593,"threads_started":1}
I20260619 13:06:58.826673 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.809512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.0.97.1:35201)
0619 13:06:58.814456 (+ 4944us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.814462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.817713 (+ 3251us) server_negotiation.cc:407] Connection header received
0619 13:06:58.817876 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.817882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.818096 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.818292 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:58.818976 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.820428 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.823437 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.824839 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.825018 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.825215 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.825583 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.825767 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4796,"thread_start_us":4654,"threads_started":1}
I20260619 13:06:58.830467 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 (DEFAULT_TABLE table=test-table [id=59a4b0d7c5504d87915c79799107507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:58.831797 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73758a037f2a435b9a8298d5011b8ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:58.837522 2016 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Bootstrap starting.
I20260619 13:06:58.841622 2016 tablet_bootstrap.cc:654] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:58.842796 2016 log.cc:824] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:58.844911 2017 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Bootstrap starting.
I20260619 13:06:58.845953 2016 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: No bootstrap required, opened a new log
I20260619 13:06:58.846228 2016 ts_tablet_manager.cc:1397] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:06:58.848338 2017 tablet_bootstrap.cc:654] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:58.848994 2019 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Bootstrap starting.
I20260619 13:06:58.849642 2017 log.cc:824] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:58.852007 2017 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: No bootstrap required, opened a new log
I20260619 13:06:58.852257 2017 ts_tablet_manager.cc:1397] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 13:06:58.852943 2019 tablet_bootstrap.cc:654] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:58.854197 2019 log.cc:824] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:58.857260 2019 tablet_bootstrap.cc:492] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: No bootstrap required, opened a new log
I20260619 13:06:58.857528 2019 ts_tablet_manager.cc:1397] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 13:06:58.857622 2016 raft_consensus.cc:348] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.858072 2016 raft_consensus.cc:374] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:58.858220 2016 raft_consensus.cc:729] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2a6cfcb6634416a713ff83de17cba7, State: Initialized, Role: FOLLOWER
I20260619 13:06:58.858924 2016 consensus_queue.cc:260] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.860579 2017 raft_consensus.cc:348] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.860915 2017 raft_consensus.cc:374] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:58.861042 2017 raft_consensus.cc:729] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b13765340c64d3fa69e51660a944231, State: Initialized, Role: FOLLOWER
I20260619 13:06:58.861646 2017 consensus_queue.cc:260] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.862660 1858 heartbeater.cc:503] Master 127.0.97.62:33445 was elected leader, sending a full tablet report...
I20260619 13:06:58.863351 2017 ts_tablet_manager.cc:1428] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 13:06:58.864319 1997 heartbeater.cc:503] Master 127.0.97.62:33445 was elected leader, sending a full tablet report...
I20260619 13:06:58.866218 2016 ts_tablet_manager.cc:1428] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260619 13:06:58.869530 2019 raft_consensus.cc:348] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.870029 2019 raft_consensus.cc:374] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:58.870137 2019 raft_consensus.cc:729] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519a6598394b405ba55d82b956e20188, State: Initialized, Role: FOLLOWER
I20260619 13:06:58.870720 2019 consensus_queue.cc:260] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.872967 2019 ts_tablet_manager.cc:1428] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260619 13:06:58.888936 1720 tablet.cc:2369] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:06:58.932670 1859 tablet.cc:2369] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:06:58.967020 1998 tablet.cc:2369] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:06:58.985513 2022 raft_consensus.cc:482] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:06:58.985868 2022 raft_consensus.cc:504] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:58.987731 2022 leader_election.cc:288] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583), 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201)
I20260619 13:06:58.994726 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.988184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57943 (local address 127.0.97.2:43583)
0619 13:06:58.988399 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.988407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.988516 (+ 109us) server_negotiation.cc:407] Connection header received
0619 13:06:58.988787 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.988795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.988888 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.989036 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:58.989642 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.990691 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.992596 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.993741 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.993896 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.994022 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.994300 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.994440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:06:58.994733 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.988140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.97.3:57943)
0619 13:06:58.988398 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.988419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.988608 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.989053 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.989058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.989092 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:58.989381 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.989399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.990838 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.990842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.992406 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.992418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.993927 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.993933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.994138 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.994162 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.994272 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.994430 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 13:06:58.995204 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.988851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34607 (local address 127.0.97.1:35201)
0619 13:06:58.989059 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:58.989065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:58.989677 (+ 612us) server_negotiation.cc:407] Connection header received
0619 13:06:58.989795 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:58.989801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:58.989881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:58.990001 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:58.990515 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.991412 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.993414 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.994265 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.994419 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.994557 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:58.994845 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:06:58.994969 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:06:58.996326 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:58.988608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35201 (local address 127.0.97.3:34607)
0619 13:06:58.989555 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:58.989572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:58.989691 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:58.990042 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:58.990046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:58.990067 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:58.990363 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.990370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.991565 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.991572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.993029 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:58.993041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:58.994501 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:58.994508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:58.994682 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:58.994706 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:06:58.994815 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:06:58.995586 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":607,"threads_started":1}
I20260619 13:06:58.996176 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73758a037f2a435b9a8298d5011b8ef5" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b13765340c64d3fa69e51660a944231" is_pre_election: true
I20260619 13:06:58.996747 1813 raft_consensus.cc:2454] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 0.
I20260619 13:06:58.997547 1880 leader_election.cc:302] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b13765340c64d3fa69e51660a944231, 9e2a6cfcb6634416a713ff83de17cba7; no voters:
I20260619 13:06:58.997666 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73758a037f2a435b9a8298d5011b8ef5" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188" is_pre_election: true
I20260619 13:06:58.998028 2022 raft_consensus.cc:2781] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:06:58.998176 2022 raft_consensus.cc:482] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:06:58.998180 1669 raft_consensus.cc:2454] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 0.
I20260619 13:06:58.998256 2022 raft_consensus.cc:3037] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.001372 2022 raft_consensus.cc:504] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:59.002080 2022 leader_election.cc:288] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583), 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201)
I20260619 13:06:59.002678 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73758a037f2a435b9a8298d5011b8ef5" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b13765340c64d3fa69e51660a944231"
I20260619 13:06:59.002708 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73758a037f2a435b9a8298d5011b8ef5" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188"
I20260619 13:06:59.002931 1669 raft_consensus.cc:3037] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.002918 1813 raft_consensus.cc:3037] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.006289 1813 raft_consensus.cc:2454] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 1.
I20260619 13:06:59.006289 1669 raft_consensus.cc:2454] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 1.
I20260619 13:06:59.006819 1880 leader_election.cc:302] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519a6598394b405ba55d82b956e20188, 9e2a6cfcb6634416a713ff83de17cba7; no voters:
I20260619 13:06:59.007150 2022 raft_consensus.cc:2781] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:59.008039 2022 raft_consensus.cc:686] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2a6cfcb6634416a713ff83de17cba7, State: Running, Role: LEADER
I20260619 13:06:59.008613 2022 consensus_queue.cc:237] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } }
I20260619 13:06:59.014010 1511 catalog_manager.cc:5495] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9e2a6cfcb6634416a713ff83de17cba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } health_report { overall_health: UNKNOWN } } }
I20260619 13:06:59.063982 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.058448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.97.3:43173)
0619 13:06:59.058675 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.058680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.058699 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:06:59.059017 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.059021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.059128 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.059261 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:59.059632 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.060359 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.061677 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.062087 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.062182 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.062799 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.063352 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.063470 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.063602 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.063668 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:06:59.072515 1669 raft_consensus.cc:1261] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Refusing update from remote peer 9e2a6cfcb6634416a713ff83de17cba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:59.072557 1813 raft_consensus.cc:1261] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Refusing update from remote peer 9e2a6cfcb6634416a713ff83de17cba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:59.073462 2026 consensus_queue.cc:1035] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.073886 2022 consensus_queue.cc:1035] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.083665 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298557620511375360 to 7298557620264091648. Current Snapshot: MvccSnapshot[applied={T|T < 7298557620511375360}]
I20260619 13:06:59.092284 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298557620511375360 to 7298557620264091648. Current Snapshot: MvccSnapshot[applied={T|T < 7298557620511375360}]
I20260619 13:06:59.102355 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:06:59.102660 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:06:59.106419 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Sending DeleteTablet for 3 replicas of tablet 73758a037f2a435b9a8298d5011b8ef5
I20260619 13:06:59.107583 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38814
I20260619 13:06:59.107556 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54992
I20260619 13:06:59.107811 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 73758a037f2a435b9a8298d5011b8ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260619 13:06:59.109851 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:06:59.110713 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:06:59.112588 2041 tablet_replica.cc:330] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: stopping tablet replica
I20260619 13:06:59.113040 2041 raft_consensus.cc:2229] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:06:59.113571 2041 raft_consensus.cc:2258] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.115857 2041 ts_tablet_manager.cc:1905] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:06:59.120151 1648 tablet_service.cc:1467] Processing CreateTablet for tablet ac2634cd71494a2497c0dc37cbab113b (DEFAULT_TABLE table=test-table [id=93525346425e46fd8500cdb35506dfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:59.120720 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2634cd71494a2497c0dc37cbab113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.120927 1787 tablet_service.cc:1467] Processing CreateTablet for tablet ac2634cd71494a2497c0dc37cbab113b (DEFAULT_TABLE table=test-table [id=93525346425e46fd8500cdb35506dfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:59.121109 2040 tablet_replica.cc:330] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: stopping tablet replica
I20260619 13:06:59.121521 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2634cd71494a2497c0dc37cbab113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.121789 2040 raft_consensus.cc:2229] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:06:59.122360 2040 raft_consensus.cc:2258] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.123226 2039 tablet_replica.cc:330] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: stopping tablet replica
I20260619 13:06:59.123368 1926 tablet_service.cc:1467] Processing CreateTablet for tablet ac2634cd71494a2497c0dc37cbab113b (DEFAULT_TABLE table=test-table [id=93525346425e46fd8500cdb35506dfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:06:59.123925 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2634cd71494a2497c0dc37cbab113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.128968 2019 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Bootstrap starting.
I20260619 13:06:59.130698 2019 tablet_bootstrap.cc:654] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.130746 2040 ts_tablet_manager.cc:1905] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:06:59.130851 2017 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Bootstrap starting.
I20260619 13:06:59.132704 2017 tablet_bootstrap.cc:654] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.135251 2016 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Bootstrap starting.
I20260619 13:06:59.123749 2039 raft_consensus.cc:2229] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:06:59.139207 2039 raft_consensus.cc:2258] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.137965 2016 tablet_bootstrap.cc:654] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.140094 2040 ts_tablet_manager.cc:1918] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:06:59.140275 2040 log.cc:1196] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/73758a037f2a435b9a8298d5011b8ef5
I20260619 13:06:59.140797 2040 ts_tablet_manager.cc:1939] T 73758a037f2a435b9a8298d5011b8ef5 P 8b13765340c64d3fa69e51660a944231: Deleting consensus metadata
I20260619 13:06:59.142799 1498 catalog_manager.cc:4841] TS 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583): tablet 73758a037f2a435b9a8298d5011b8ef5 (table test-table [id=59a4b0d7c5504d87915c79799107507c]) successfully deleted
I20260619 13:06:59.144187 2041 ts_tablet_manager.cc:1918] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:06:59.144368 2041 log.cc:1196] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/73758a037f2a435b9a8298d5011b8ef5
I20260619 13:06:59.145032 2041 ts_tablet_manager.cc:1939] T 73758a037f2a435b9a8298d5011b8ef5 P 519a6598394b405ba55d82b956e20188: Deleting consensus metadata
I20260619 13:06:59.147018 1498 catalog_manager.cc:4841] TS 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201): tablet 73758a037f2a435b9a8298d5011b8ef5 (table test-table [id=59a4b0d7c5504d87915c79799107507c]) successfully deleted
I20260619 13:06:59.148726 2016 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: No bootstrap required, opened a new log
I20260619 13:06:59.149158 2016 ts_tablet_manager.cc:1397] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20260619 13:06:59.149556 2017 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: No bootstrap required, opened a new log
I20260619 13:06:59.149729 2017 ts_tablet_manager.cc:1397] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.001s
I20260619 13:06:59.150065 2016 raft_consensus.cc:348] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.150302 2016 raft_consensus.cc:374] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.150410 2016 raft_consensus.cc:729] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2a6cfcb6634416a713ff83de17cba7, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.150516 2017 raft_consensus.cc:348] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.150740 2017 raft_consensus.cc:374] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.150832 2017 raft_consensus.cc:729] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b13765340c64d3fa69e51660a944231, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.150709 2016 consensus_queue.cc:260] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.151142 2017 consensus_queue.cc:260] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.151791 2019 tablet_bootstrap.cc:492] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: No bootstrap required, opened a new log
I20260619 13:06:59.151839 2017 ts_tablet_manager.cc:1428] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:06:59.152014 2019 ts_tablet_manager.cc:1397] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
I20260619 13:06:59.152479 2016 ts_tablet_manager.cc:1428] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:06:59.153427 2039 ts_tablet_manager.cc:1905] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:06:59.153357 2019 raft_consensus.cc:348] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.153715 2019 raft_consensus.cc:374] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.153853 2019 raft_consensus.cc:729] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519a6598394b405ba55d82b956e20188, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.154217 2019 consensus_queue.cc:260] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.155844 2019 ts_tablet_manager.cc:1428] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:06:59.160470 2039 ts_tablet_manager.cc:1918] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:06:59.160651 2039 log.cc:1196] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/73758a037f2a435b9a8298d5011b8ef5
I20260619 13:06:59.161121 2039 ts_tablet_manager.cc:1939] T 73758a037f2a435b9a8298d5011b8ef5 P 9e2a6cfcb6634416a713ff83de17cba7: Deleting consensus metadata
I20260619 13:06:59.162655 1499 catalog_manager.cc:4841] TS 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3:43173): tablet 73758a037f2a435b9a8298d5011b8ef5 (table test-table [id=59a4b0d7c5504d87915c79799107507c]) successfully deleted
I20260619 13:06:59.219022 2036 raft_consensus.cc:482] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:06:59.219233 2036 raft_consensus.cc:504] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.220323 2036 leader_election.cc:288] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201), 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583)
I20260619 13:06:59.220849 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2634cd71494a2497c0dc37cbab113b" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188" is_pre_election: true
I20260619 13:06:59.220957 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2634cd71494a2497c0dc37cbab113b" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b13765340c64d3fa69e51660a944231" is_pre_election: true
I20260619 13:06:59.221102 1669 raft_consensus.cc:2454] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 0.
I20260619 13:06:59.221220 1813 raft_consensus.cc:2454] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 0.
I20260619 13:06:59.221566 1880 leader_election.cc:302] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519a6598394b405ba55d82b956e20188, 9e2a6cfcb6634416a713ff83de17cba7; no voters:
I20260619 13:06:59.221891 2036 raft_consensus.cc:2781] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:06:59.222018 2036 raft_consensus.cc:482] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:06:59.222108 2036 raft_consensus.cc:3037] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.224202 2036 raft_consensus.cc:504] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.224674 2036 leader_election.cc:288] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 election: Requested vote from peers 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201), 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583)
I20260619 13:06:59.225349 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2634cd71494a2497c0dc37cbab113b" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b13765340c64d3fa69e51660a944231"
I20260619 13:06:59.225329 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2634cd71494a2497c0dc37cbab113b" candidate_uuid: "9e2a6cfcb6634416a713ff83de17cba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188"
I20260619 13:06:59.225565 1813 raft_consensus.cc:3037] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.225580 1669 raft_consensus.cc:3037] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.227895 1669 raft_consensus.cc:2454] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 1.
I20260619 13:06:59.228343 1880 leader_election.cc:302] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519a6598394b405ba55d82b956e20188, 9e2a6cfcb6634416a713ff83de17cba7; no voters:
I20260619 13:06:59.228614 2036 raft_consensus.cc:2781] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:59.228621 1813 raft_consensus.cc:2454] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2a6cfcb6634416a713ff83de17cba7 in term 1.
I20260619 13:06:59.228787 2036 raft_consensus.cc:686] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2a6cfcb6634416a713ff83de17cba7, State: Running, Role: LEADER
I20260619 13:06:59.229079 2036 consensus_queue.cc:237] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.232259 1511 catalog_manager.cc:5495] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9e2a6cfcb6634416a713ff83de17cba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } health_report { overall_health: HEALTHY } } }
Jun 19 13:06:59 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:06:59.303074 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.288069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33445 (local address 127.0.0.1:39628)
0619 13:06:59.288739 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.288767 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.288944 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.289349 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.289353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.290148 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:59.290805 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.290816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.291900 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.291908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.292847 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.292859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.293626 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.293638 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.293816 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.294508 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:59.294545 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:59.296954 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:59.300014 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:59.300030 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:59.300040 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:59.300527 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:59.300870 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:59.300877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:59.300880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:59.301012 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:59.301555 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:59.301563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:59.302178 (+ 615us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.302342 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.302649 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":351,"threads_started":1}
I20260619 13:06:59.303156 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.288160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.0.97.62:33445)
0619 13:06:59.289106 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.289110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.289125 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:06:59.289164 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.289167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.289215 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.289324 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.290984 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.291775 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.292990 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.293401 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.293532 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.297179 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.297184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.297187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.297214 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.299878 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.300650 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.300656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.300660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.300772 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.301108 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.301116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.301119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.301431 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.301570 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.302425 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.302779 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":809,"thread_start_us":658,"threads_started":1}
I20260619 13:06:59.308355 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39628:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:06:59.309376 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:06:59.316072 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 129a7d0ade8347b88585eeeb79969cb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3eaa34b2819348b1a82c2886fbbe7bd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:06:59.316097 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 129a7d0ade8347b88585eeeb79969cb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3eaa34b2819348b1a82c2886fbbe7bd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:06:59.316592 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129a7d0ade8347b88585eeeb79969cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.316593 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129a7d0ade8347b88585eeeb79969cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.322081 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 129a7d0ade8347b88585eeeb79969cb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3eaa34b2819348b1a82c2886fbbe7bd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:06:59.322599 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129a7d0ade8347b88585eeeb79969cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:59.323168 2019 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: Bootstrap starting.
I20260619 13:06:59.324843 2019 tablet_bootstrap.cc:654] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.325804 2017 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: Bootstrap starting.
I20260619 13:06:59.327688 2017 tablet_bootstrap.cc:654] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.330372 2016 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: Bootstrap starting.
I20260619 13:06:59.331779 2016 tablet_bootstrap.cc:654] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:59.334483 2017 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: No bootstrap required, opened a new log
I20260619 13:06:59.334554 2019 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: No bootstrap required, opened a new log
I20260619 13:06:59.334731 2019 ts_tablet_manager.cc:1397] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20260619 13:06:59.334774 2017 ts_tablet_manager.cc:1397] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260619 13:06:59.334933 2016 tablet_bootstrap.cc:492] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: No bootstrap required, opened a new log
I20260619 13:06:59.335110 2016 ts_tablet_manager.cc:1397] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 13:06:59.335490 2019 raft_consensus.cc:348] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.335809 2016 raft_consensus.cc:348] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.335947 2019 raft_consensus.cc:374] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.336026 2019 raft_consensus.cc:729] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519a6598394b405ba55d82b956e20188, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.336064 2016 raft_consensus.cc:374] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.336136 2016 raft_consensus.cc:729] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2a6cfcb6634416a713ff83de17cba7, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.336306 2019 consensus_queue.cc:260] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.336390 2016 consensus_queue.cc:260] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.336786 2036 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.336760 2024 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.337069 2036 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.337127 2024 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.337631 2016 ts_tablet_manager.cc:1428] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 13:06:59.338184 2019 ts_tablet_manager.cc:1428] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 13:06:59.335714 2017 raft_consensus.cc:348] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.339612 2017 raft_consensus.cc:374] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:06:59.339697 2017 raft_consensus.cc:729] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b13765340c64d3fa69e51660a944231, State: Initialized, Role: FOLLOWER
I20260619 13:06:59.339998 2017 consensus_queue.cc:260] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.340497 2023 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.340867 2023 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.341331 2017 ts_tablet_manager.cc:1428] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260619 13:06:59.513686 2023 raft_consensus.cc:482] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:06:59.513921 2023 raft_consensus.cc:504] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.515635 2023 leader_election.cc:288] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201), 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3:43173)
I20260619 13:06:59.535181 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.516342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44923 (local address 127.0.97.3:43173)
0619 13:06:59.516608 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.516617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.528560 (+ 11943us) server_negotiation.cc:407] Connection header received
0619 13:06:59.528718 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.528722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.528793 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.528883 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:59.529505 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.530705 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.533342 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.534207 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.534345 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.534448 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.534737 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.534866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 13:06:59.535241 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.516097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43173 (local address 127.0.97.2:44923)
0619 13:06:59.528443 (+ 12346us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.528461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.528589 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.528910 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.528914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.528940 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:59.529336 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.529350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.530843 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.530848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.533194 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.533204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.534380 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.534387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.534582 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.534609 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.534713 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.534861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12230,"thread_start_us":11657,"threads_started":1}
I20260619 13:06:59.535754 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.515918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35201 (local address 127.0.97.2:55495)
0619 13:06:59.528117 (+ 12199us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.528157 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.528342 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.528751 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.528755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.528777 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:06:59.529055 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.529065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.530335 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.530342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.532777 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.532795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.534622 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.534630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.534807 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.534865 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.534961 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.535110 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12027,"thread_start_us":12185,"threads_started":1}
I20260619 13:06:59.535751 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.516812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55495 (local address 127.0.97.1:35201)
0619 13:06:59.527510 (+ 10698us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.527520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.528371 (+ 851us) server_negotiation.cc:407] Connection header received
0619 13:06:59.528485 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.528496 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.528585 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.528699 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:06:59.529254 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.530202 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.532973 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.534198 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.534329 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.534434 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.535311 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.535448 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10513,"thread_start_us":10708,"threads_started":1}
I20260619 13:06:59.536653 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129a7d0ade8347b88585eeeb79969cb3" candidate_uuid: "8b13765340c64d3fa69e51660a944231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2a6cfcb6634416a713ff83de17cba7" is_pre_election: true
I20260619 13:06:59.536790 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129a7d0ade8347b88585eeeb79969cb3" candidate_uuid: "8b13765340c64d3fa69e51660a944231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188" is_pre_election: true
I20260619 13:06:59.537062 1669 raft_consensus.cc:2454] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b13765340c64d3fa69e51660a944231 in term 0.
I20260619 13:06:59.537128 1952 raft_consensus.cc:2454] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b13765340c64d3fa69e51660a944231 in term 0.
I20260619 13:06:59.537689 1741 leader_election.cc:302] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519a6598394b405ba55d82b956e20188, 8b13765340c64d3fa69e51660a944231; no voters:
I20260619 13:06:59.538129 2023 raft_consensus.cc:2781] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:06:59.538259 2023 raft_consensus.cc:482] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:06:59.538398 2023 raft_consensus.cc:3037] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.540833 2023 raft_consensus.cc:504] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.541611 2023 leader_election.cc:288] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [CANDIDATE]: Term 1 election: Requested vote from peers 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201), 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3:43173)
I20260619 13:06:59.542001 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129a7d0ade8347b88585eeeb79969cb3" candidate_uuid: "8b13765340c64d3fa69e51660a944231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519a6598394b405ba55d82b956e20188"
I20260619 13:06:59.542253 1669 raft_consensus.cc:3037] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.544499 1669 raft_consensus.cc:2454] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b13765340c64d3fa69e51660a944231 in term 1.
I20260619 13:06:59.544916 1741 leader_election.cc:302] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519a6598394b405ba55d82b956e20188, 8b13765340c64d3fa69e51660a944231; no voters:
I20260619 13:06:59.544939 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129a7d0ade8347b88585eeeb79969cb3" candidate_uuid: "8b13765340c64d3fa69e51660a944231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2a6cfcb6634416a713ff83de17cba7"
I20260619 13:06:59.545506 1952 raft_consensus.cc:3037] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:59.547703 1952 raft_consensus.cc:2454] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b13765340c64d3fa69e51660a944231 in term 1.
I20260619 13:06:59.547964 2023 raft_consensus.cc:2781] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:59.548477 2023 raft_consensus.cc:686] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [term 1 LEADER]: Becoming Leader. State: Replica: 8b13765340c64d3fa69e51660a944231, State: Running, Role: LEADER
I20260619 13:06:59.549001 2023 consensus_queue.cc:237] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } }
I20260619 13:06:59.551632 2059 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: TxnStatusTablet state changed. Reason: New leader 8b13765340c64d3fa69e51660a944231. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.552233 2059 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:06:59.552961 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:06:59.553171 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:06:59.553418 1511 catalog_manager.cc:5495] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b13765340c64d3fa69e51660a944231 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } health_report { overall_health: UNKNOWN } } }
I20260619 13:06:59.585131 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.574067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.0.1:55000)
0619 13:06:59.574295 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.574316 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.574453 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.578824 (+ 4371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.578829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.578844 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:06:59.579115 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.579132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.580467 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.580476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.582017 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.582031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.583043 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.583049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.583238 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.583301 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.584475 (+ 1174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.584501 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.584620 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.584788 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 13:06:59.585182 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.574406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.97.2:43583)
0619 13:06:59.578076 (+ 3670us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.578081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.578100 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:06:59.578180 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.578253 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.578350 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.578816 (+ 466us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:59.579284 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.580324 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.582179 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.582870 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.583025 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.583549 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.584357 (+ 808us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.584523 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.584640 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.584733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3556,"thread_start_us":3407,"threads_started":1}
I20260619 13:06:59.589962 1669 raft_consensus.cc:1261] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Refusing update from remote peer 8b13765340c64d3fa69e51660a944231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:59.590153 1952 raft_consensus.cc:1261] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Refusing update from remote peer 8b13765340c64d3fa69e51660a944231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:59.590696 2059 consensus_queue.cc:1035] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.591163 2023 consensus_queue.cc:1035] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.601204 2036 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: TxnStatusTablet state changed. Reason: New leader 8b13765340c64d3fa69e51660a944231. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.601578 2036 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.605626 2024 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: TxnStatusTablet state changed. Reason: New leader 8b13765340c64d3fa69e51660a944231. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.605978 2024 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.607036 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298557622632132608 to 7298557622477357056. Current Snapshot: MvccSnapshot[applied={T|T < 7298557622632132608}]
I20260619 13:06:59.610298 2059 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.610668 2059 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:06:59.610939 2059 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.611210 2059 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 8b13765340c64d3fa69e51660a944231: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:06:59.619061 2026 consensus_queue.cc:1035] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.622977 2036 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.623363 2036 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 9e2a6cfcb6634416a713ff83de17cba7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.647450 2026 consensus_queue.cc:1035] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:59.660445 2024 tablet_replica.cc:439] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b13765340c64d3fa69e51660a944231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519a6598394b405ba55d82b956e20188" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35201 } } peers { permanent_uuid: "8b13765340c64d3fa69e51660a944231" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "9e2a6cfcb6634416a713ff83de17cba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43173 } } }
I20260619 13:06:59.660728 2024 tablet_replica.cc:442] T 129a7d0ade8347b88585eeeb79969cb3 P 519a6598394b405ba55d82b956e20188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:06:59.667765 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.655644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57243 (local address 127.0.97.2:43583)
0619 13:06:59.655893 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.655902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.661284 (+ 5382us) server_negotiation.cc:407] Connection header received
0619 13:06:59.661700 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.661705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.661784 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.661845 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:59.662581 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.663657 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.665509 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.666034 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.666174 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.666600 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.666696 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.666787 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.666902 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.666982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:06:59.667773 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.649708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.97.3:57243)
0619 13:06:59.661183 (+ 11475us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.661204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.661455 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.662059 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.662066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.662081 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:06:59.662397 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.662409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.663841 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.663846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.665369 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.665386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.666212 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.666219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.666363 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.666427 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.666789 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.666802 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.666879 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.667011 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11245,"thread_start_us":15303,"threads_started":1}
I20260619 13:06:59.719202 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.711674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42187 (local address 127.0.97.3:43173)
0619 13:06:59.711869 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.711876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.712494 (+ 618us) server_negotiation.cc:407] Connection header received
0619 13:06:59.712712 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.712717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.712829 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.712961 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:59.713580 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.714685 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.716900 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.717611 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.717720 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.718376 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.718526 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.718589 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.718816 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.718903 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:06:59.720577 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.711641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43173 (local address 127.0.97.2:42187)
0619 13:06:59.712362 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:59.712383 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:59.712552 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:59.712979 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:59.712984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:59.712996 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:06:59.713326 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.713334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.714871 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.714877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.716733 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:59.716752 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.717911 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.717919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:59.718100 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.718175 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:06:59.718672 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:06:59.718682 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:06:59.719527 (+ 845us) client_negotiation.cc:240] Negotiation successful
0619 13:06:59.719683 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":387,"threads_started":1}
I20260619 13:06:59.773248 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:06:59.773543 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:06:59.776801 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 700be010de2a4bb3a603dfcde2bfdee4: Sending DeleteTablet for 3 replicas of tablet ac2634cd71494a2497c0dc37cbab113b
I20260619 13:06:59.777627 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2634cd71494a2497c0dc37cbab113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260619 13:06:59.777773 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2634cd71494a2497c0dc37cbab113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54992
I20260619 13:06:59.777879 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2634cd71494a2497c0dc37cbab113b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:06:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38814
I20260619 13:06:59.788419 2086 tablet_replica.cc:330] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: stopping tablet replica
I20260619 13:06:59.788939 2086 raft_consensus.cc:2229] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:06:59.789392 2086 raft_consensus.cc:2258] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.791030 2086 ts_tablet_manager.cc:1905] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:06:59.793977 2087 tablet_replica.cc:330] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: stopping tablet replica
I20260619 13:06:59.794189 2087 raft_consensus.cc:2229] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:06:59.794569 2087 raft_consensus.cc:2258] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.796173 2087 ts_tablet_manager.cc:1905] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:06:59.801559 2086 ts_tablet_manager.cc:1918] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:06:59.801749 2086 log.cc:1196] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/ac2634cd71494a2497c0dc37cbab113b
I20260619 13:06:59.802253 2086 ts_tablet_manager.cc:1939] T ac2634cd71494a2497c0dc37cbab113b P 8b13765340c64d3fa69e51660a944231: Deleting consensus metadata
I20260619 13:06:59.803480 1498 catalog_manager.cc:4841] TS 8b13765340c64d3fa69e51660a944231 (127.0.97.2:43583): tablet ac2634cd71494a2497c0dc37cbab113b (table test-table [id=93525346425e46fd8500cdb35506dfc7]) successfully deleted
I20260619 13:06:59.803990 2087 ts_tablet_manager.cc:1918] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:06:59.804139 2087 log.cc:1196] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/ac2634cd71494a2497c0dc37cbab113b
I20260619 13:06:59.804555 2087 ts_tablet_manager.cc:1939] T ac2634cd71494a2497c0dc37cbab113b P 9e2a6cfcb6634416a713ff83de17cba7: Deleting consensus metadata
I20260619 13:06:59.805692 1499 catalog_manager.cc:4841] TS 9e2a6cfcb6634416a713ff83de17cba7 (127.0.97.3:43173): tablet ac2634cd71494a2497c0dc37cbab113b (table test-table [id=93525346425e46fd8500cdb35506dfc7]) successfully deleted
I20260619 13:06:59.806095 2085 tablet_replica.cc:330] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: stopping tablet replica
I20260619 13:06:59.806296 2085 raft_consensus.cc:2229] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:06:59.806569 2085 raft_consensus.cc:2258] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:06:59.807564 2085 ts_tablet_manager.cc:1905] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:06:59 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:06:59.815034 2085 ts_tablet_manager.cc:1918] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:06:59.815178 2085 log.cc:1196] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/ac2634cd71494a2497c0dc37cbab113b
I20260619 13:06:59.815677 2085 ts_tablet_manager.cc:1939] T ac2634cd71494a2497c0dc37cbab113b P 519a6598394b405ba55d82b956e20188: Deleting consensus metadata
I20260619 13:06:59.816905 1498 catalog_manager.cc:4841] TS 519a6598394b405ba55d82b956e20188 (127.0.97.1:35201): tablet ac2634cd71494a2497c0dc37cbab113b (table test-table [id=93525346425e46fd8500cdb35506dfc7]) successfully deleted
I20260619 13:06:59.818707 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.797700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.97.1:35201)
0619 13:06:59.797903 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.797913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.801897 (+ 3984us) server_negotiation.cc:407] Connection header received
0619 13:06:59.802582 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.802589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.802678 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.802762 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.804819 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.806269 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.807851 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.808285 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.808444 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.812377 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.812388 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.812406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.812445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.815627 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.816387 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.816396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.816400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.816551 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.817027 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.817032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.817036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.817470 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.817822 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.818097 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.818341 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 13:06:59.819940 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50872
I20260619 13:06:59.840771 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.823315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.97.62:33445)
0619 13:06:59.826016 (+ 2701us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.826022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.826042 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:06:59.826100 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.826105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.826179 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.826291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.827943 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.829091 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.830469 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.831127 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.831255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.833462 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.833469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.833475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.833516 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.836890 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.838139 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.838148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.838160 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.838466 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.839057 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.839062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.839065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.839520 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.839801 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.840183 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.840378 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2568,"thread_start_us":2416,"threads_started":1}
W20260619 13:06:59.841650 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39634
I20260619 13:06:59.843945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1583
I20260619 13:06:59.858256 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.844838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.0.97.1:35201)
0619 13:06:59.845059 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.845068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.845490 (+ 422us) server_negotiation.cc:407] Connection header received
0619 13:06:59.845621 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.845626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.845710 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.845808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.846964 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.848041 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.849328 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.849971 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.850096 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.853013 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.853018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.853022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.853056 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.855535 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.856498 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.856502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.856505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.856633 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.857012 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.857016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.857018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.857259 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.857433 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.857782 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.857962 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 13:06:59.858853 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50878
W20260619 13:06:59.859406 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:06:59.859613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1583
I20260619 13:06:59.877882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1723
Jun 19 13:06:59 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:06:59.894412 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.878615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.0.97.2:43583)
0619 13:06:59.878823 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.878830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.878958 (+ 128us) server_negotiation.cc:407] Connection header received
0619 13:06:59.879166 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.879171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.879246 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.879360 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.880686 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.881546 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.882795 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.883242 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.883375 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.888210 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.888223 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.888238 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.888281 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.891526 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.892239 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.892246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.892249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.892389 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.892812 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.892821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.892824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.893218 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.893585 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.893827 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.894053 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 13:06:59.895229 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55008
W20260619 13:06:59.895895 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:06:59.896064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1723
I20260619 13:06:59.915364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1862
Jun 19 13:06:59 dist-test-slave-q11v krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:06:59.929742 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:59.916223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.3:43173)
0619 13:06:59.916402 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:59.916407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:59.916425 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:06:59.916583 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:59.916586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:59.916649 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:59.916732 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:59.918028 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.918703 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.919907 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:59.920312 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:59.920404 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:59.924480 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:59.924495 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:59.924512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:59.924549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:59.927312 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.927922 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.927932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.927936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.928081 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:59.928394 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:59.928397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:59.928399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:59.928741 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:59.928960 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:59.929147 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 13:06:59.929318 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 13:06:59.930368 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38832
W20260619 13:06:59.930986 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:06:59.931141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1862
I20260619 13:06:59.952575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 1480
I20260619 13:06:59.953840 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:00.067284 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 1480
2026-06-19T13:07:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4808 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:00 dist-test-slave-q11v krb5kdc[2097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:00 dist-test-slave-q11v krb5kdc[2097](info): set up 2 sockets
Jun 19 13:07:00 dist-test-slave-q11v krb5kdc[2097](info): commencing operation
krb5kdc: starting...
W20260619 13:07:01.161890 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:01 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:01.354339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45177
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:01.689145 2110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:01.689467 2110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:01.689678 2110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:01.697044 2110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:01.697167 2110 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:07:01.697265 2110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:01.697371 2110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:01.697446 2110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:01.697531 2110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:01.708184 2110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45177
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:01.709527 2110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:01.710917 2110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:01.724732 2115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:01.724756 2116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:01.725250 2118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:01.725970 2110 server_base.cc:1034] running on GCE node
I20260619 13:07:01.726723 2110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:01.728179 2110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:01.729426 2110 hybrid_clock.cc:648] HybridClock initialized: now 1781874421729368 us; error 42 us; skew 500 ppm
I20260619 13:07:01.729823 2110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:01 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:01.732899 2110 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:01.734481 2110 webserver.cc:460] Webserver started at http://127.0.97.62:37723/ using document root <none> and password file <none>
I20260619 13:07:01.735205 2110 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:01.735338 2110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:01.735656 2110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:01.738763 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "ff799081eb9044b4a02eb296152f45ee"
format_stamp: "Formatted at 2026-06-19 13:07:01 on dist-test-slave-q11v"
server_key: "63dc40f4b01ce4f96f67e023dcfb36fa"
server_key_iv: "6e2b947b01a17de9fe1076ed93a7d39e"
server_key_version: "encryptionkey@0"
I20260619 13:07:01.739552 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff799081eb9044b4a02eb296152f45ee"
format_stamp: "Formatted at 2026-06-19 13:07:01 on dist-test-slave-q11v"
server_key: "63dc40f4b01ce4f96f67e023dcfb36fa"
server_key_iv: "6e2b947b01a17de9fe1076ed93a7d39e"
server_key_version: "encryptionkey@0"
I20260619 13:07:01.744911 2110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:07:01.748569 2125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:01.750087 2110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:07:01.750289 2110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "ff799081eb9044b4a02eb296152f45ee"
format_stamp: "Formatted at 2026-06-19 13:07:01 on dist-test-slave-q11v"
server_key: "63dc40f4b01ce4f96f67e023dcfb36fa"
server_key_iv: "6e2b947b01a17de9fe1076ed93a7d39e"
server_key_version: "encryptionkey@0"
I20260619 13:07:01.750528 2110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:01.773078 2110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:01.777539 2110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:01.777848 2110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:01.799351 2110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45177
I20260619 13:07:01.799372 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45177 every 8 connection(s)
I20260619 13:07:01.800757 2110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:01.803304 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2110
I20260619 13:07:01.803525 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:01.804299 388 external_mini_cluster.cc:1428] Setting key 49f66ade9a36ced3454dca09f6d11cd0
I20260619 13:07:01.805277 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:01.817320 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Bootstrap starting.
Jun 19 13:07:01 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:01.820801 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:01.821962 2187 log.cc:824] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:01.824807 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: No bootstrap required, opened a new log
I20260619 13:07:01.825915 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:01.806408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.0.97.62:45177)
0619 13:07:01.807140 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:01.807153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:01.807201 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:07:01.808741 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:01.808765 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:01.809169 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:01.809688 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:01.811138 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:01.812445 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:01.813678 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:01.814378 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:01.814575 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:01.818644 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:01.818661 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:01.818675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:01.818718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:01.821857 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:01.822626 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:01.822632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:01.822638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:01.822804 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:01.823179 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:01.823186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:01.823190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:01.823688 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:01.823986 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:01.824390 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 13:07:01.824805 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260619 13:07:01.832578 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } }
I20260619 13:07:01.832983 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:01.833086 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff799081eb9044b4a02eb296152f45ee, State: Initialized, Role: FOLLOWER
I20260619 13:07:01.833667 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } }
I20260619 13:07:01.833880 2187 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:01.833981 2187 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:01.834105 2187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:01.836480 2187 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } }
I20260619 13:07:01.836951 2187 leader_election.cc:302] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff799081eb9044b4a02eb296152f45ee; no voters:
I20260619 13:07:01.837445 2187 leader_election.cc:288] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:01.837584 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:01.838290 2192 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [term 1 LEADER]: Becoming Leader. State: Replica: ff799081eb9044b4a02eb296152f45ee, State: Running, Role: LEADER
I20260619 13:07:01.838864 2192 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } }
I20260619 13:07:01.839604 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:01.841987 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff799081eb9044b4a02eb296152f45ee. Latest consensus state: current_term: 1 leader_uuid: "ff799081eb9044b4a02eb296152f45ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } } }
I20260619 13:07:01.842154 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff799081eb9044b4a02eb296152f45ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff799081eb9044b4a02eb296152f45ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45177 } } }
I20260619 13:07:01.842453 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:01.842482 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee [sys.catalog]: This master's current role is: LEADER
W20260619 13:07:01.857523 2208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:07:01.857717 2208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:07:01.857856 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:01.860080 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:01.869211 2200 catalog_manager.cc:1347] Generated new cluster ID: 4ca5340e3b8f444ba1ccd6b4fd21bc94
I20260619 13:07:01.869349 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:01.882194 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:01.883937 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:01.890882 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Generated new TSK 0
I20260619 13:07:01.891743 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:01.943459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45177
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:02.274763 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:02.275059 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:02.275162 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:02.282742 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:02.282850 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:02.282935 2213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:07:02.282987 2213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:02.283056 2213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:02.292863 2213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:02.294229 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:02.295545 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:02.308482 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:02.308522 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:02.308501 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:02.309695 2213 server_base.cc:1034] running on GCE node
I20260619 13:07:02.310323 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:02.311271 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:02.312471 2213 hybrid_clock.cc:648] HybridClock initialized: now 1781874422312400 us; error 60 us; skew 500 ppm
I20260619 13:07:02.312747 2213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:02 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:02.315922 2213 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:02.317351 2213 webserver.cc:460] Webserver started at http://127.0.97.1:35107/ using document root <none> and password file <none>
I20260619 13:07:02.318060 2213 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:02.318168 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:02.318421 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:02.321355 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "4e75a89f6a5c7e6c000911175b36dc85"
server_key_iv: "8827263ef6a23e4fce12e1c45e21ccdf"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.322137 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "4e75a89f6a5c7e6c000911175b36dc85"
server_key_iv: "8827263ef6a23e4fce12e1c45e21ccdf"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.327275 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:07:02.330735 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:02.332129 2213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:07:02.332298 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "4e75a89f6a5c7e6c000911175b36dc85"
server_key_iv: "8827263ef6a23e4fce12e1c45e21ccdf"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.332494 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:02.352735 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:02.357481 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:02.357813 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:02.360414 2213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:02.360546 2213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:02.360685 2213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:02.360773 2213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:07:02 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:02.387423 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.361275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54799 (local address 127.0.97.62:45177)
0619 13:07:02.361757 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:02.361763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:02.363168 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 13:07:02.366525 (+ 3357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:02.366534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:02.366644 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:02.366788 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:02.369031 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.370127 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.372656 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.373445 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.373653 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.378529 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:02.378539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:02.378547 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:02.378604 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:02.382525 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.383621 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.383628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.383634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.383767 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.384290 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.384299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.384306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.384755 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:02.384946 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:02.386548 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0619 13:07:02.386872 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":219,"threads_started":1}
I20260619 13:07:02.387490 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.362006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.1:54799)
0619 13:07:02.362972 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:02.363026 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:02.366052 (+ 3026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:02.367063 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:02.367071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:02.367890 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:02.368829 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.368864 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.370363 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.370371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.372473 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.372495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.373749 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.373756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.373953 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.374847 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:02.374895 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:02.378197 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:02.382751 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.382761 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.382785 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.383474 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.383890 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.383895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.383899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.384132 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.384903 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:02.384915 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:02.385431 (+ 516us) client_negotiation.cc:769] Sending connection context
0619 13:07:02.385874 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 13:07:02.386342 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":410,"threads_started":1}
I20260619 13:07:02.406519 2213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34115
I20260619 13:07:02.406574 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34115 every 8 connection(s)
I20260619 13:07:02.408004 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:02.410241 2349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45177
I20260619 13:07:02.410475 2349 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:02.411304 2349 heartbeater.cc:511] Master 127.0.97.62:45177 requested a full tablet report, sending...
I20260619 13:07:02.411974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2213
I20260619 13:07:02.412073 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.394476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44843 (local address 127.0.97.62:45177)
0619 13:07:02.394639 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:02.394643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:02.395443 (+ 800us) server_negotiation.cc:407] Connection header received
0619 13:07:02.395801 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:02.395811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:02.395903 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:02.395982 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:02.397956 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.398706 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.401467 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.402277 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.402436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.405054 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:02.405059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:02.405062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:02.405118 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:02.408318 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.409805 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.409812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.409818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.409969 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.410440 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.410444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.410448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.410827 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:02.410983 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:02.411494 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 13:07:02.411733 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:07:02.412160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:02.412263 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.394488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.1:44843)
0619 13:07:02.395313 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:02.395340 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:02.395515 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:02.396255 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:02.396260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:02.397179 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:02.397763 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.397788 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.398898 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.398906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.401233 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.401254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.402523 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.402532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.402814 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.403592 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:02.403629 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:02.404885 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:02.408451 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.408459 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.408465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.409711 (+ 1246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.410123 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.410127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.410130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.410281 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.410976 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:02.410980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:02.411228 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 13:07:02.411358 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 13:07:02.411725 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":21,"thread_start_us":434,"threads_started":1}
I20260619 13:07:02.412925 388 external_mini_cluster.cc:1428] Setting key 645f82b5407654462a233b3d711cf6af
I20260619 13:07:02.413574 2142 ts_manager.cc:194] Registered new tserver with Master: c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115)
I20260619 13:07:02.415115 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54799
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:02.452924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45177
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:02.781445 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:02.781762 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:02.781863 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:02.789269 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:02.789381 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:02.789464 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:07:02.789516 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:02.789610 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:02.799616 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:02.800858 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:02.802212 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:02.816025 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:02.816092 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:02.816045 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:02.816783 2353 server_base.cc:1034] running on GCE node
I20260619 13:07:02.817437 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:02.818419 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:02.819636 2353 hybrid_clock.cc:648] HybridClock initialized: now 1781874422819577 us; error 59 us; skew 500 ppm
I20260619 13:07:02.819952 2353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:02 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:02.823122 2353 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:02.824659 2353 webserver.cc:460] Webserver started at http://127.0.97.2:45073/ using document root <none> and password file <none>
I20260619 13:07:02.825335 2353 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:02.825470 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:02.825824 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:02.828644 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "1d42d46e8f0a41b3af2dbbf0410499f8"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "bf56b7f37c76d30c23c3b4e7adec92c5"
server_key_iv: "4ae152bb39a6ec746d59ca2952f99be3"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.829411 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1d42d46e8f0a41b3af2dbbf0410499f8"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "bf56b7f37c76d30c23c3b4e7adec92c5"
server_key_iv: "4ae152bb39a6ec746d59ca2952f99be3"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.834599 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:07:02.838017 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:02.839389 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:07:02.839551 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "1d42d46e8f0a41b3af2dbbf0410499f8"
format_stamp: "Formatted at 2026-06-19 13:07:02 on dist-test-slave-q11v"
server_key: "bf56b7f37c76d30c23c3b4e7adec92c5"
server_key_iv: "4ae152bb39a6ec746d59ca2952f99be3"
server_key_version: "encryptionkey@0"
I20260619 13:07:02.839761 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:02.879421 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:02.883371 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:02.883673 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:02.886382 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:02.886509 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:02.886657 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:02.886775 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:07:02 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:02.914049 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.886978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50951 (local address 127.0.97.62:45177)
0619 13:07:02.887226 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:02.887233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:02.889013 (+ 1780us) server_negotiation.cc:407] Connection header received
0619 13:07:02.892430 (+ 3417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:02.892439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:02.892541 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:02.892669 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:02.895401 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.896121 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.899219 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.899624 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.899817 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.906006 (+ 6189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:02.906016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:02.906025 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:02.906082 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:02.909852 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.911042 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.911049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.911055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.911183 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.911771 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.911774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.911776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.912039 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:02.912144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:02.913286 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0619 13:07:02.913551 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 13:07:02.915946 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.887592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.2:50951)
0619 13:07:02.888828 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:02.888893 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:02.892046 (+ 3153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:02.893008 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:02.893021 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:02.894057 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:02.895149 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.895205 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.896299 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.896307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.899073 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.899087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.899939 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.899951 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.900333 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.901321 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:02.901368 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:02.905630 (+ 4262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:02.910012 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.910026 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.910052 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.910878 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.911292 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.911300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.911306 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.911645 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.912160 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:02.912173 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:02.912768 (+ 595us) client_negotiation.cc:769] Sending connection context
0619 13:07:02.913155 (+ 387us) client_negotiation.cc:240] Negotiation successful
0619 13:07:02.913931 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":712,"threads_started":1}
I20260619 13:07:02.935883 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.922244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.2:34435)
0619 13:07:02.923011 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:02.923030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:02.923192 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:02.923596 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:02.923611 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:02.924363 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:02.924838 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.924855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.925959 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.925963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.926975 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:02.926986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.927714 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.927722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:02.927876 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.928484 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:02.928500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:02.929193 (+ 693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:02.932704 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.932713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.932719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.933424 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.933971 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:02.933976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:02.933980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:02.934101 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:02.934927 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:02.934930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:02.935081 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 13:07:02.935188 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:07:02.935452 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":479,"threads_started":1}
I20260619 13:07:02.936136 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:02.922110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34435 (local address 127.0.97.62:45177)
0619 13:07:02.922264 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:02.922268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:02.923116 (+ 848us) server_negotiation.cc:407] Connection header received
0619 13:07:02.923335 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:02.923340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:02.923428 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:02.923532 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:02.924998 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.925811 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.927113 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:02.927525 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:02.927649 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:02.929425 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:02.929430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:02.929433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:02.929463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:02.932507 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.933648 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.933656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.933662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.933828 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:02.934212 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:02.934220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:02.934227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:02.934802 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:02.934976 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:02.935331 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 13:07:02.935644 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 13:07:02.945645 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35969
I20260619 13:07:02.945693 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35969 every 8 connection(s)
I20260619 13:07:02.946954 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:02.948930 2488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45177
I20260619 13:07:02.949146 2488 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:02.949862 2488 heartbeater.cc:511] Master 127.0.97.62:45177 requested a full tablet report, sending...
I20260619 13:07:02.951128 2142 ts_manager.cc:194] Registered new tserver with Master: 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969)
I20260619 13:07:02.951928 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50951
I20260619 13:07:02.951960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2353
I20260619 13:07:02.952124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:02.952636 388 external_mini_cluster.cc:1428] Setting key 957c9dd9565cf92609e99ecd87c6b8ef
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:02.992789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45177
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:03.314838 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:03.315133 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:03.315300 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:03.322656 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:03.322762 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:03.322926 2492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:07:03.323004 2492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:03.323133 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:03.332818 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:03.334139 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:03.335481 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:03.349642 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:03.349614 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:03.349785 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:03.350512 2492 server_base.cc:1034] running on GCE node
I20260619 13:07:03.351202 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:03.352175 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:03.353380 2492 hybrid_clock.cc:648] HybridClock initialized: now 1781874423353323 us; error 46 us; skew 500 ppm
I20260619 13:07:03.353765 2492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:03 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:03.360494 2492 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:03.362128 2492 webserver.cc:460] Webserver started at http://127.0.97.3:35737/ using document root <none> and password file <none>
I20260619 13:07:03.362835 2492 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:03.362972 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:03.363296 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:03.366377 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
format_stamp: "Formatted at 2026-06-19 13:07:03 on dist-test-slave-q11v"
server_key: "cc23c0fede6d86bb6f8ac112eba5a0e6"
server_key_iv: "415d71a1c4a2e7517881a6d22742b810"
server_key_version: "encryptionkey@0"
I20260619 13:07:03.367189 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
format_stamp: "Formatted at 2026-06-19 13:07:03 on dist-test-slave-q11v"
server_key: "cc23c0fede6d86bb6f8ac112eba5a0e6"
server_key_iv: "415d71a1c4a2e7517881a6d22742b810"
server_key_version: "encryptionkey@0"
I20260619 13:07:03.372605 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:03.376611 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:03.378156 2492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:03.378356 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
format_stamp: "Formatted at 2026-06-19 13:07:03 on dist-test-slave-q11v"
server_key: "cc23c0fede6d86bb6f8ac112eba5a0e6"
server_key_iv: "415d71a1c4a2e7517881a6d22742b810"
server_key_version: "encryptionkey@0"
I20260619 13:07:03.378571 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:03.399458 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:03.403961 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:03.404273 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:03.407354 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:03.407480 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:03.407629 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:03.407732 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:03.417899 2349 heartbeater.cc:503] Master 127.0.97.62:45177 was elected leader, sending a full tablet report...
Jun 19 13:07:03 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:03.432922 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.407976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38725 (local address 127.0.97.62:45177)
0619 13:07:03.408163 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.408170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.409785 (+ 1615us) server_negotiation.cc:407] Connection header received
0619 13:07:03.412398 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.412404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.412500 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.412613 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:03.415215 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.416302 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.419312 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.420071 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.420218 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.425676 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:03.425682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:03.425685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:03.425716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:03.429178 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.430102 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.430110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.430115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.430279 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.430953 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.430959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.430964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.431371 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:03.431527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.432335 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.432557 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:07:03.434247 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.408672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.3:38725)
0619 13:07:03.409585 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.409661 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.412122 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.413074 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.413090 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.414014 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:03.414938 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.414974 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.416468 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.416473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.419131 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.419143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.420484 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.420495 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.420950 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.421956 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:03.422006 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:03.425387 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:03.429318 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.429327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.429345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.429875 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.430397 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.430407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.430411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.430781 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.431503 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:03.431515 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:03.431900 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.432177 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.432740 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260619 13:07:03.456799 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.440360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38289 (local address 127.0.97.62:45177)
0619 13:07:03.440525 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.440530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.442055 (+ 1525us) server_negotiation.cc:407] Connection header received
0619 13:07:03.442251 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.442256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.442329 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.442427 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:03.443575 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.444762 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.446888 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.447315 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.447445 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.449509 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:03.449523 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:03.449533 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:03.449565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:03.453198 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.454466 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.454475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.454480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.454628 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.455129 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.455134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.455138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.455459 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:03.455594 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.456382 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.456559 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 13:07:03.457295 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.441050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.97.3:38289)
0619 13:07:03.441943 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.441964 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.442128 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.442451 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.442455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.442969 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:03.443407 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.443421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.444884 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.444890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.446739 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.446750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.447506 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.447512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.447730 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.448522 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:03.448546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:03.449315 (+ 769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:03.453380 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.453388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.453395 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.454309 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.454780 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.454788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.454795 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.454988 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.455626 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:03.455634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:03.455897 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.456074 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.456530 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":583,"threads_started":1}
I20260619 13:07:03.464977 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34475
I20260619 13:07:03.465008 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34475 every 8 connection(s)
I20260619 13:07:03.466285 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:03.468392 2627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45177
I20260619 13:07:03.468665 2627 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:03.469352 2627 heartbeater.cc:511] Master 127.0.97.62:45177 requested a full tablet report, sending...
I20260619 13:07:03.470795 2142 ts_manager.cc:194] Registered new tserver with Master: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:03.471498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2492
I20260619 13:07:03.471658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:03.471625 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38725
I20260619 13:07:03.472196 388 external_mini_cluster.cc:1428] Setting key e609ead4f447ac9145a0eb38c18f8acc
I20260619 13:07:03.475014 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:07:03 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:07:03 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:03.521844 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.507899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.0.97.62:45177)
0619 13:07:03.508173 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.508182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.508529 (+ 347us) server_negotiation.cc:407] Connection header received
0619 13:07:03.508745 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.508749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.508811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.508911 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:03.510376 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.511115 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.512341 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.512757 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.512888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.516766 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:03.516770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:03.516773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:03.516804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:03.519341 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.519936 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.519943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.519947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.520069 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.520417 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.520422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.520425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.520689 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:03.520942 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.521250 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.521462 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 13:07:03.525805 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:03.532430 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:03.576164 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.546330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.0.97.3:34475)
0619 13:07:03.561755 (+ 15425us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.561765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.568936 (+ 7171us) server_negotiation.cc:407] Connection header received
0619 13:07:03.569057 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.569067 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.569318 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.569500 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:03.570455 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571751 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.573697 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.574918 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.575003 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.575178 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.575655 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.575812 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15212,"thread_start_us":15434,"threads_started":1}
I20260619 13:07:03.576261 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.546013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34475 (local address 127.0.0.1:35448)
0619 13:07:03.568599 (+ 22586us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.568619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.568774 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.569845 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.569850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.569879 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:03.570251 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.570268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571871 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.571878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.573481 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.573502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.575301 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.575306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.575435 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.575465 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.575610 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.575804 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22462,"mutex_wait_us":3,"spinlock_wait_cycles":9600,"thread_start_us":21142,"threads_started":1}
I20260619 13:07:03.577049 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.545264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34115 (local address 127.0.0.1:35746)
0619 13:07:03.561513 (+ 16249us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.561621 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.561910 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.568711 (+ 6801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.568717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.568764 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:03.569324 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.569337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571453 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.571460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.574069 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.574081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.576235 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.576242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.576418 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.576449 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.576538 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.576678 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15816,"mutex_wait_us":126,"spinlock_wait_cycles":7808,"thread_start_us":15627,"threads_started":1}
I20260619 13:07:03.577771 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.545515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.0.97.1:34115)
0619 13:07:03.568118 (+ 22603us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.568127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.568154 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:07:03.568251 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.568262 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.568490 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.568710 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:03.569903 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571334 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.574253 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.575745 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.575877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.576087 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.577290 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.577455 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22458,"thread_start_us":22342,"threads_started":1}
I20260619 13:07:03.578164 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.545504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35969 (local address 127.0.0.1:60684)
0619 13:07:03.567431 (+ 21927us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.567451 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.567618 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.568646 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.568656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.568687 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:03.569240 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.569260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571269 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.571274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.574456 (+ 3182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.574468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.577385 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.577391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.577553 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.577625 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.577716 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.577841 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21809,"mutex_wait_us":146,"thread_start_us":22843,"threads_started":1}
I20260619 13:07:03.578248 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.546355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.0.97.2:35969)
0619 13:07:03.566938 (+ 20583us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.566952 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.567571 (+ 619us) server_negotiation.cc:407] Connection header received
0619 13:07:03.567801 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.567822 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.568139 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.568362 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:03.569502 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.571125 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.574644 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.575940 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.576052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.576239 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.577783 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.577962 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20416,"thread_start_us":20290,"threads_started":1}
I20260619 13:07:03.580633 2557 tablet_service.cc:1467] Processing CreateTablet for tablet cb245231ec504ca986feac4192507c1e (DEFAULT_TABLE table=test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:03.581159 2418 tablet_service.cc:1467] Processing CreateTablet for tablet cb245231ec504ca986feac4192507c1e (DEFAULT_TABLE table=test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:03.581655 2278 tablet_service.cc:1467] Processing CreateTablet for tablet cb245231ec504ca986feac4192507c1e (DEFAULT_TABLE table=test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:03.582228 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb245231ec504ca986feac4192507c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.582376 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb245231ec504ca986feac4192507c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.582868 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb245231ec504ca986feac4192507c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.593953 2646 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Bootstrap starting.
I20260619 13:07:03.596529 2647 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Bootstrap starting.
I20260619 13:07:03.597927 2646 tablet_bootstrap.cc:654] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.598030 2648 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Bootstrap starting.
I20260619 13:07:03.599634 2646 log.cc:824] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:03.600086 2647 tablet_bootstrap.cc:654] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.600879 2648 tablet_bootstrap.cc:654] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.601290 2647 log.cc:824] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:03.602058 2648 log.cc:824] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:03.603605 2646 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: No bootstrap required, opened a new log
I20260619 13:07:03.603888 2646 ts_tablet_manager.cc:1397] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:07:03.604156 2647 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: No bootstrap required, opened a new log
I20260619 13:07:03.604430 2647 ts_tablet_manager.cc:1397] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:07:03.604575 2648 tablet_bootstrap.cc:492] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: No bootstrap required, opened a new log
I20260619 13:07:03.604791 2648 ts_tablet_manager.cc:1397] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:07:03.612082 2648 raft_consensus.cc:348] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.612687 2648 raft_consensus.cc:374] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.612843 2648 raft_consensus.cc:729] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.613518 2648 consensus_queue.cc:260] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.614647 2627 heartbeater.cc:503] Master 127.0.97.62:45177 was elected leader, sending a full tablet report...
I20260619 13:07:03.614663 2647 raft_consensus.cc:348] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.614614 2646 raft_consensus.cc:348] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.615048 2647 raft_consensus.cc:374] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.615057 2646 raft_consensus.cc:374] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.615154 2647 raft_consensus.cc:729] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.615154 2646 raft_consensus.cc:729] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.615737 2647 consensus_queue.cc:260] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.615739 2646 consensus_queue.cc:260] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.616699 2648 ts_tablet_manager.cc:1428] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 13:07:03.618700 2647 ts_tablet_manager.cc:1428] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 13:07:03.620483 2488 heartbeater.cc:503] Master 127.0.97.62:45177 was elected leader, sending a full tablet report...
I20260619 13:07:03.621142 2646 ts_tablet_manager.cc:1428] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260619 13:07:03.660544 2350 tablet.cc:2369] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:03.698886 2489 tablet.cc:2369] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:03.717905 2628 tablet.cc:2369] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:03.762903 2653 raft_consensus.cc:482] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:03.763300 2653 raft_consensus.cc:504] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.765075 2653 leader_election.cc:288] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:03.772081 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.765682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33169 (local address 127.0.97.3:34475)
0619 13:07:03.765859 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.765864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.766867 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 13:07:03.767053 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.767059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.767119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.767205 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:03.767798 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.768476 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.770347 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.771106 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.771253 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.771374 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.771697 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.771819 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:07:03.772095 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.765465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56799 (local address 127.0.97.1:34115)
0619 13:07:03.765711 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.765719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.766707 (+ 988us) server_negotiation.cc:407] Connection header received
0619 13:07:03.766930 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.766934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.767001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.767097 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:03.767697 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.768417 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.770356 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.771106 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.771260 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.771378 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.771715 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.771835 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:07:03.772279 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.765555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34475 (local address 127.0.97.2:33169)
0619 13:07:03.766752 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.766770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.766917 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.767233 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.767237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.767276 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:03.767643 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.767651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.768618 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.768624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.770193 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.770205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.771348 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.771360 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.771513 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.771561 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.771671 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.771807 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":29,"thread_start_us":841,"threads_started":1}
I20260619 13:07:03.772310 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.765471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34115 (local address 127.0.97.2:56799)
0619 13:07:03.766582 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.766600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.766783 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.767114 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.767118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.767142 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:03.767543 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.767552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.768548 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.768556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.770206 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.770218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.771378 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.771386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.771564 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.771586 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.771686 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.771809 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":90,"thread_start_us":737,"threads_started":1}
I20260619 13:07:03.773412 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb245231ec504ca986feac4192507c1e" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" is_pre_election: true
I20260619 13:07:03.773407 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb245231ec504ca986feac4192507c1e" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" is_pre_election: true
I20260619 13:07:03.773942 2299 raft_consensus.cc:2454] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:03.773941 2577 raft_consensus.cc:2454] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:03.774602 2370 leader_election.cc:302] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:03.775007 2653 raft_consensus.cc:2781] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:03.775143 2653 raft_consensus.cc:482] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:03.775255 2653 raft_consensus.cc:3037] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.777441 2653 raft_consensus.cc:504] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.778115 2653 leader_election.cc:288] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Requested vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:03.778637 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb245231ec504ca986feac4192507c1e" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
I20260619 13:07:03.778637 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb245231ec504ca986feac4192507c1e" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
I20260619 13:07:03.778837 2577 raft_consensus.cc:3037] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.778856 2299 raft_consensus.cc:3037] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.780980 2299 raft_consensus.cc:2454] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:03.781004 2577 raft_consensus.cc:2454] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:03.781418 2370 leader_election.cc:302] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:03.781749 2653 raft_consensus.cc:2781] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:03.782207 2653 raft_consensus.cc:686] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Running, Role: LEADER
I20260619 13:07:03.782756 2653 consensus_queue.cc:237] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.787541 2141 catalog_manager.cc:5495] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } health_report { overall_health: UNKNOWN } } }
Jun 19 13:07:03 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:03.845460 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.831994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45177 (local address 127.0.0.1:48492)
0619 13:07:03.832569 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.832585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.832751 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.833079 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.833083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.833833 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:03.834105 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.834114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.835074 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.835081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.836179 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.836190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.836988 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.836996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.837170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.837881 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:03.837900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:03.840067 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:03.842951 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.842967 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.842973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.843461 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.843832 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:03.843835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:03.843837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:03.844001 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:03.844595 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:03.844602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:03.844755 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.844903 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.845138 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
I20260619 13:07:03.845505 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.831951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.97.62:45177)
0619 13:07:03.832116 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.832120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.832674 (+ 554us) server_negotiation.cc:407] Connection header received
0619 13:07:03.832870 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.832879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.832927 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.833018 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:03.834241 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.834956 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.836300 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.836773 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.836926 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.840256 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:03.840261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:03.840264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:03.840292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:03.842826 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.843564 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.843567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.843570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.843701 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:03.844098 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:03.844101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:03.844103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:03.844445 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:03.844566 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.844994 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.845269 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":56}
I20260619 13:07:03.850507 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:07:03.851620 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:03.859532 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 5831783d58b34ab4911b04fa5d7aaa48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5172eddab07141df9d8e78894121db21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:03.859591 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 5831783d58b34ab4911b04fa5d7aaa48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5172eddab07141df9d8e78894121db21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:03.860073 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 5831783d58b34ab4911b04fa5d7aaa48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5172eddab07141df9d8e78894121db21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:03.860553 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5831783d58b34ab4911b04fa5d7aaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.860553 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5831783d58b34ab4911b04fa5d7aaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.860750 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5831783d58b34ab4911b04fa5d7aaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:03.866485 2648 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Bootstrap starting.
I20260619 13:07:03.867676 2647 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Bootstrap starting.
I20260619 13:07:03.867734 2646 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: Bootstrap starting.
I20260619 13:07:03.867949 2648 tablet_bootstrap.cc:654] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.868991 2647 tablet_bootstrap.cc:654] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.869429 2646 tablet_bootstrap.cc:654] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:03.870329 2648 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: No bootstrap required, opened a new log
I20260619 13:07:03.870504 2648 ts_tablet_manager.cc:1397] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:07:03.871122 2647 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: No bootstrap required, opened a new log
I20260619 13:07:03.871228 2647 ts_tablet_manager.cc:1397] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:07:03.871493 2648 raft_consensus.cc:348] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.871733 2647 raft_consensus.cc:348] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.871830 2648 raft_consensus.cc:374] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.871896 2647 raft_consensus.cc:374] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.871953 2647 raft_consensus.cc:729] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.871930 2648 raft_consensus.cc:729] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.872175 2647 consensus_queue.cc:260] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.872794 2647 ts_tablet_manager.cc:1428] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 13:07:03.872680 2654 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.873045 2654 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.872926 2648 consensus_queue.cc:260] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.873495 2652 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.873883 2652 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.874277 2648 ts_tablet_manager.cc:1428] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260619 13:07:03.876194 2646 tablet_bootstrap.cc:492] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: No bootstrap required, opened a new log
I20260619 13:07:03.876339 2646 ts_tablet_manager.cc:1397] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 13:07:03.877002 2646 raft_consensus.cc:348] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.877184 2646 raft_consensus.cc:374] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:03.877286 2646 raft_consensus.cc:729] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Initialized, Role: FOLLOWER
I20260619 13:07:03.877548 2646 consensus_queue.cc:260] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.877837 2653 raft_consensus.cc:482] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:03.878024 2653 raft_consensus.cc:504] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.878427 2657 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.878760 2653 leader_election.cc:288] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:03.878871 2657 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.878968 2646 ts_tablet_manager.cc:1428] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:03.879433 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5831783d58b34ab4911b04fa5d7aaa48" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" is_pre_election: true
I20260619 13:07:03.879478 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5831783d58b34ab4911b04fa5d7aaa48" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" is_pre_election: true
I20260619 13:07:03.879679 2577 raft_consensus.cc:2454] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:03.879766 2299 raft_consensus.cc:2454] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:03.880167 2371 leader_election.cc:302] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, 1d42d46e8f0a41b3af2dbbf0410499f8; no voters:
I20260619 13:07:03.880443 2657 raft_consensus.cc:2781] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:03.880563 2657 raft_consensus.cc:482] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:03.880728 2657 raft_consensus.cc:3037] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.883289 2657 raft_consensus.cc:504] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.883900 2657 leader_election.cc:288] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Requested vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:03.884266 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5831783d58b34ab4911b04fa5d7aaa48" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
I20260619 13:07:03.884344 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5831783d58b34ab4911b04fa5d7aaa48" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
I20260619 13:07:03.884449 2299 raft_consensus.cc:3037] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.884600 2577 raft_consensus.cc:3037] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:03.886397 2299 raft_consensus.cc:2454] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:03.886754 2577 raft_consensus.cc:2454] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:03.886794 2370 leader_election.cc:302] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:03.887173 2657 raft_consensus.cc:2781] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:03.887338 2657 raft_consensus.cc:686] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Running, Role: LEADER
I20260619 13:07:03.887575 2657 consensus_queue.cc:237] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:03.889431 2653 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: TxnStatusTablet state changed. Reason: New leader 1d42d46e8f0a41b3af2dbbf0410499f8. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.889730 2653 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:03.890385 2141 catalog_manager.cc:5495] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:03.891745 2669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:07:03.892045 2669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:07:03.911147 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.905317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.97.2:35969)
0619 13:07:03.906162 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.906167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.906180 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:07:03.906274 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.906277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.906349 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.906501 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:03.906838 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.907599 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.908833 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.909301 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.909424 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.909952 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:03.910514 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:03.910677 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.910759 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.910836 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729}
I20260619 13:07:03.911190 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.905316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35969 (local address 127.0.0.1:60690)
0619 13:07:03.905752 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.905768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.905872 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.906514 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.906519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.906530 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:03.906714 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.906722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.907720 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.907727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.908697 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.908708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.909453 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.909459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.909699 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.909754 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:03.910631 (+ 877us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:03.910652 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.910739 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.910887 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260619 13:07:03.919293 2299 raft_consensus.cc:1261] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:03.919338 2577 raft_consensus.cc:1261] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:03.920233 2653 consensus_queue.cc:1035] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:03.920588 2657 consensus_queue.cc:1035] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:03.932021 2654 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: TxnStatusTablet state changed. Reason: New leader 1d42d46e8f0a41b3af2dbbf0410499f8. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.932328 2654 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.932534 2652 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: TxnStatusTablet state changed. Reason: New leader 1d42d46e8f0a41b3af2dbbf0410499f8. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.932811 2652 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.934348 2672 mvcc.cc:205] Tried to move back new op lower bound from 7298557640363253760 to 7298557640245760000. Current Snapshot: MvccSnapshot[applied={T|T < 7298557640363253760}]
I20260619 13:07:03.934872 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298557640363253760 to 7298557640245760000. Current Snapshot: MvccSnapshot[applied={T|T < 7298557640363253760}]
I20260619 13:07:03.953687 2652 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.954038 2652 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.957244 2654 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.957495 2654 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P c4d0b3e55ce941f0ae40c7cd03bc2e68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:03.963121 2657 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.963461 2657 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:03.963923 2657 tablet_replica.cc:439] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } } }
I20260619 13:07:03.964162 2657 tablet_replica.cc:442] T 5831783d58b34ab4911b04fa5d7aaa48 P 1d42d46e8f0a41b3af2dbbf0410499f8: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:03.976190 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.970456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.97.2:35969)
0619 13:07:03.970630 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.970636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.970867 (+ 231us) server_negotiation.cc:407] Connection header received
0619 13:07:03.971045 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.971052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.971135 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.971249 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:03.971581 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.972822 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.974306 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.974963 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.975088 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.975455 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:03.975602 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:03.975686 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.975804 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.975881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:07:03.995235 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.988490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34153 (local address 127.0.97.2:35969)
0619 13:07:03.988703 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:03.988709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:03.989471 (+ 762us) server_negotiation.cc:407] Connection header received
0619 13:07:03.989668 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:03.989674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:03.989752 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:03.989850 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:03.990526 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.991674 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.993139 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.993864 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.994000 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.994426 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:03.994601 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:03.994687 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:03.994814 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 13:07:03.994900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":53}
I20260619 13:07:03.995277 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:03.988490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35969 (local address 127.0.97.2:34153)
0619 13:07:03.989358 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:03.989376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:03.989521 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:03.989891 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:03.989894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:03.989904 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:03.990349 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.990361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.991811 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.991815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.993001 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:03.993012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:03.994058 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:03.994062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:03.994180 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:03.994236 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:03.994705 (+ 469us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:03.994713 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:07:03.994778 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 13:07:03.994894 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":3,"thread_start_us":480,"threads_started":1}
I20260619 13:07:04.004680 2299 raft_consensus.cc:1261] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:04.004735 2577 raft_consensus.cc:1261] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:04.005209 2657 consensus_queue.cc:1035] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:04.005748 2653 consensus_queue.cc:1035] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:04.011222 2673 mvcc.cc:205] Tried to move back new op lower bound from 7298557640718835712 to 7298557639818764288. Current Snapshot: MvccSnapshot[applied={T|T < 7298557640718835712}]
I20260619 13:07:04.068027 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:04.068349 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:04.072157 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Sending DeleteTablet for 3 replicas of tablet cb245231ec504ca986feac4192507c1e
I20260619 13:07:04.073290 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet cb245231ec504ca986feac4192507c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260619 13:07:04.073318 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet cb245231ec504ca986feac4192507c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35448
I20260619 13:07:04.073326 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet cb245231ec504ca986feac4192507c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60684
I20260619 13:07:04.076149 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:04.077020 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:04.085251 2691 tablet_replica.cc:330] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: stopping tablet replica
I20260619 13:07:04.085769 2691 raft_consensus.cc:2229] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:04.086164 2691 raft_consensus.cc:2258] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:04.089689 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 8712e07166f249528f91ea8a4e27a99e (DEFAULT_TABLE table=test-table [id=59d93e3f64314081a7b8d87c97b7a402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:04.089798 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 8712e07166f249528f91ea8a4e27a99e (DEFAULT_TABLE table=test-table [id=59d93e3f64314081a7b8d87c97b7a402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:04.090225 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8712e07166f249528f91ea8a4e27a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:04.090515 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8712e07166f249528f91ea8a4e27a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:04.091370 2691 ts_tablet_manager.cc:1905] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:04.096865 2648 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Bootstrap starting.
I20260619 13:07:04.098579 2648 tablet_bootstrap.cc:654] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:04.099849 2646 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Bootstrap starting.
I20260619 13:07:04.102190 2646 tablet_bootstrap.cc:654] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:04.105736 2691 ts_tablet_manager.cc:1918] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:04.107041 2691 log.cc:1196] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/cb245231ec504ca986feac4192507c1e
I20260619 13:07:04.107656 2691 ts_tablet_manager.cc:1939] T cb245231ec504ca986feac4192507c1e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting consensus metadata
I20260619 13:07:04.109743 2648 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: No bootstrap required, opened a new log
I20260619 13:07:04.109777 2127 catalog_manager.cc:4841] TS c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115): tablet cb245231ec504ca986feac4192507c1e (table test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]) successfully deleted
I20260619 13:07:04.110050 2648 ts_tablet_manager.cc:1397] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260619 13:07:04.110925 2648 raft_consensus.cc:348] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.111280 2648 raft_consensus.cc:374] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:04.111400 2648 raft_consensus.cc:729] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, State: Initialized, Role: FOLLOWER
I20260619 13:07:04.111742 2648 consensus_queue.cc:260] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.114078 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 8712e07166f249528f91ea8a4e27a99e (DEFAULT_TABLE table=test-table [id=59d93e3f64314081a7b8d87c97b7a402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:04.114722 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8712e07166f249528f91ea8a4e27a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:04.114930 2648 ts_tablet_manager.cc:1428] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 13:07:04.123700 2646 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: No bootstrap required, opened a new log
I20260619 13:07:04.123958 2646 ts_tablet_manager.cc:1397] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20260619 13:07:04.125068 2646 raft_consensus.cc:348] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.125409 2646 raft_consensus.cc:374] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:04.125510 2646 raft_consensus.cc:729] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Initialized, Role: FOLLOWER
I20260619 13:07:04.125907 2646 consensus_queue.cc:260] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.127027 2689 tablet_replica.cc:330] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: stopping tablet replica
I20260619 13:07:04.129451 2689 raft_consensus.cc:2229] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:04.129909 2689 raft_consensus.cc:2258] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:04.129936 2646 ts_tablet_manager.cc:1428] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260619 13:07:04.131805 2647 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Bootstrap starting.
I20260619 13:07:04.133538 2647 tablet_bootstrap.cc:654] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:04.134860 2689 ts_tablet_manager.cc:1905] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:04.135564 2690 tablet_replica.cc:330] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: stopping tablet replica
I20260619 13:07:04.136042 2690 raft_consensus.cc:2229] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:04.136690 2690 raft_consensus.cc:2258] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:04.136894 2647 tablet_bootstrap.cc:492] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: No bootstrap required, opened a new log
I20260619 13:07:04.137122 2647 ts_tablet_manager.cc:1397] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 13:07:04.137980 2647 raft_consensus.cc:348] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.138304 2647 raft_consensus.cc:374] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:04.138409 2690 ts_tablet_manager.cc:1905] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:04.138417 2647 raft_consensus.cc:729] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Initialized, Role: FOLLOWER
I20260619 13:07:04.138758 2647 consensus_queue.cc:260] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.144723 2689 ts_tablet_manager.cc:1918] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:04.144919 2689 log.cc:1196] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/cb245231ec504ca986feac4192507c1e
I20260619 13:07:04.145406 2689 ts_tablet_manager.cc:1939] T cb245231ec504ca986feac4192507c1e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting consensus metadata
I20260619 13:07:04.147246 2128 catalog_manager.cc:4841] TS 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475): tablet cb245231ec504ca986feac4192507c1e (table test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]) successfully deleted
I20260619 13:07:04.147305 2690 ts_tablet_manager.cc:1918] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:04.147459 2690 log.cc:1196] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/cb245231ec504ca986feac4192507c1e
I20260619 13:07:04.148015 2690 ts_tablet_manager.cc:1939] T cb245231ec504ca986feac4192507c1e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting consensus metadata
I20260619 13:07:04.149312 2647 ts_tablet_manager.cc:1428] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260619 13:07:04.149621 2126 catalog_manager.cc:4841] TS 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969): tablet cb245231ec504ca986feac4192507c1e (table test-table [id=752bb0578d4d4d81b5dd0a79daf255bd]) successfully deleted
I20260619 13:07:04.202178 2654 raft_consensus.cc:482] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:04.202435 2654 raft_consensus.cc:504] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.204137 2654 leader_election.cc:288] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:04.213685 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.204588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55307 (local address 127.0.97.3:34475)
0619 13:07:04.204899 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:04.204910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:04.206062 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 13:07:04.206268 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:04.206272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:04.206340 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:04.206428 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:04.207303 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.208385 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.211102 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.212311 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.212414 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.212511 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:04.213220 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 13:07:04.213365 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 13:07:04.213905 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.204555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35969 (local address 127.0.97.1:50737)
0619 13:07:04.205580 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:04.205626 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:04.205848 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:04.206559 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:04.206562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:04.206613 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:04.206966 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.206975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.208573 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.208579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.210600 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.210613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.212610 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.212619 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.212903 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.213072 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 13:07:04.213170 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:07:04.213399 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":166,"spinlock_wait_cycles":2816,"thread_start_us":628,"threads_started":1}
I20260619 13:07:04.214223 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.204577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34475 (local address 127.0.97.1:55307)
0619 13:07:04.205946 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:04.205967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:04.206113 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:04.206470 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:04.206475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:04.206502 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:04.207056 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.207067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.208567 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.208574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.210931 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.210948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.212645 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.212650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.212905 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.213071 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 13:07:04.213170 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:07:04.213399 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":183,"thread_start_us":1031,"threads_started":1}
I20260619 13:07:04.214917 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.204980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50737 (local address 127.0.97.2:35969)
0619 13:07:04.205176 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:04.205182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:04.205731 (+ 549us) server_negotiation.cc:407] Connection header received
0619 13:07:04.206039 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:04.206046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:04.206130 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:04.206243 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:04.207174 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.208403 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.210782 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.212113 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.212283 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.212424 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:04.214155 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0619 13:07:04.214301 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:07:04.214915 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8712e07166f249528f91ea8a4e27a99e" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" is_pre_election: true
I20260619 13:07:04.215229 2577 raft_consensus.cc:2454] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 0.
I20260619 13:07:04.215767 2231 leader_election.cc:302] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:04.215718 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8712e07166f249528f91ea8a4e27a99e" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" is_pre_election: true
I20260619 13:07:04.216248 2654 raft_consensus.cc:2781] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:04.216174 2438 raft_consensus.cc:2454] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 0.
I20260619 13:07:04.216357 2654 raft_consensus.cc:482] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:04.216444 2654 raft_consensus.cc:3037] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:04.218819 2654 raft_consensus.cc:504] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.219426 2654 leader_election.cc:288] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 election: Requested vote from peers 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:04.219868 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8712e07166f249528f91ea8a4e27a99e" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8"
I20260619 13:07:04.220064 2438 raft_consensus.cc:3037] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:04.220431 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8712e07166f249528f91ea8a4e27a99e" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
I20260619 13:07:04.220634 2577 raft_consensus.cc:3037] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:04.223502 2577 raft_consensus.cc:2454] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 1.
I20260619 13:07:04.223518 2438 raft_consensus.cc:2454] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 1.
I20260619 13:07:04.223964 2229 leader_election.cc:302] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:04.224332 2654 raft_consensus.cc:2781] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:04.224862 2654 raft_consensus.cc:686] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 LEADER]: Becoming Leader. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Running, Role: LEADER
I20260619 13:07:04.225430 2654 consensus_queue.cc:237] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:04.229118 2142 catalog_manager.cc:5495] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:04.252863 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.246608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.97.1:34115)
0619 13:07:04.246783 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:04.246790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:04.246961 (+ 171us) server_negotiation.cc:407] Connection header received
0619 13:07:04.247190 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:04.247193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:04.247275 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:04.247395 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:04.247817 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.248573 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.250520 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.250919 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.251031 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.251588 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:04.252178 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:04.252301 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:04.252459 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 13:07:04.252529 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:07:04.269333 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.263765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60045 (local address 127.0.97.2:35969)
0619 13:07:04.263941 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:04.263945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:04.264461 (+ 516us) server_negotiation.cc:407] Connection header received
0619 13:07:04.264687 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:04.264691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:04.264754 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:04.264849 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:04.265310 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.266161 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.267544 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.267987 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.268146 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.268561 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:04.268681 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:04.268781 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:04.268914 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 13:07:04.268996 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:07:04.269423 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.263760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35969 (local address 127.0.97.1:60045)
0619 13:07:04.264357 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:04.264373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:04.264543 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:04.264864 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:04.264869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:04.264881 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:04.265164 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.265172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.266318 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.266323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.267398 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.267409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.268182 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.268188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.268328 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.268380 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:04.268790 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:04.268803 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:07:04.268892 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:07:04.269042 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":343,"threads_started":1}
I20260619 13:07:04.279543 2577 raft_consensus.cc:1261] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Refusing update from remote peer c4d0b3e55ce941f0ae40c7cd03bc2e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:04.279911 2438 raft_consensus.cc:1261] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Refusing update from remote peer c4d0b3e55ce941f0ae40c7cd03bc2e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:04.280231 2697 consensus_queue.cc:1035] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:04.280649 2654 consensus_queue.cc:1035] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:04.313045 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.306395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38393 (local address 127.0.97.1:34115)
0619 13:07:04.306559 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:04.306568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:04.306784 (+ 216us) server_negotiation.cc:407] Connection header received
0619 13:07:04.306945 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:04.306951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:04.307042 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:04.307152 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:04.307713 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.308839 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.310954 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.311612 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.311735 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.312343 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:04.312473 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:04.312559 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:04.312740 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 13:07:04.312804 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:07:04.313200 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:04.306458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34115 (local address 127.0.97.2:38393)
0619 13:07:04.306693 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:04.306709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:04.306814 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:04.307315 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:04.307322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:04.307342 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:04.307542 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.307554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.309001 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.309007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.310761 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:04.310777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:04.311873 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:04.311880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:04.312083 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:04.312146 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:04.312609 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:04.312619 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:07:04.312716 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:07:04.312865 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
W20260619 13:07:04.314282 2674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:07:05.919232 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:05.919483 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:05.923903 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Sending DeleteTablet for 3 replicas of tablet 8712e07166f249528f91ea8a4e27a99e
I20260619 13:07:05.924623 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 8712e07166f249528f91ea8a4e27a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260619 13:07:05.924945 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 8712e07166f249528f91ea8a4e27a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60684
I20260619 13:07:05.924934 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8712e07166f249528f91ea8a4e27a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35448
I20260619 13:07:05.935904 2731 tablet_replica.cc:330] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: stopping tablet replica
I20260619 13:07:05.936344 2731 raft_consensus.cc:2229] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:05.936681 2731 raft_consensus.cc:2258] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:05.937832 2731 ts_tablet_manager.cc:1905] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:05.938035 2730 tablet_replica.cc:330] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: stopping tablet replica
I20260619 13:07:05.938256 2730 raft_consensus.cc:2229] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:05.938503 2730 raft_consensus.cc:2258] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:05.938992 2729 tablet_replica.cc:330] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: stopping tablet replica
I20260619 13:07:05.939193 2729 raft_consensus.cc:2229] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:05.939658 2729 raft_consensus.cc:2258] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:05.940371 2730 ts_tablet_manager.cc:1905] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:05.942209 2729 ts_tablet_manager.cc:1905] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:05.943162 2731 ts_tablet_manager.cc:1918] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:05.943310 2731 log.cc:1196] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/8712e07166f249528f91ea8a4e27a99e
I20260619 13:07:05.943735 2731 ts_tablet_manager.cc:1939] T 8712e07166f249528f91ea8a4e27a99e P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting consensus metadata
I20260619 13:07:05.944926 2128 catalog_manager.cc:4841] TS 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475): tablet 8712e07166f249528f91ea8a4e27a99e (table test-table [id=59d93e3f64314081a7b8d87c97b7a402]) successfully deleted
I20260619 13:07:05.947484 2730 ts_tablet_manager.cc:1918] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:05.947628 2730 log.cc:1196] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/8712e07166f249528f91ea8a4e27a99e
I20260619 13:07:05.948061 2730 ts_tablet_manager.cc:1939] T 8712e07166f249528f91ea8a4e27a99e P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting consensus metadata
I20260619 13:07:05.949149 2126 catalog_manager.cc:4841] TS 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969): tablet 8712e07166f249528f91ea8a4e27a99e (table test-table [id=59d93e3f64314081a7b8d87c97b7a402]) successfully deleted
I20260619 13:07:05.950397 2729 ts_tablet_manager.cc:1918] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:05.950539 2729 log.cc:1196] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/8712e07166f249528f91ea8a4e27a99e
I20260619 13:07:05.950990 2729 ts_tablet_manager.cc:1939] T 8712e07166f249528f91ea8a4e27a99e P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting consensus metadata
I20260619 13:07:05.952133 2127 catalog_manager.cc:4841] TS c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115): tablet 8712e07166f249528f91ea8a4e27a99e (table test-table [id=59d93e3f64314081a7b8d87c97b7a402]) successfully deleted
Jun 19 13:07:05 dist-test-slave-q11v krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:07:05 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:05.991115 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:05.977072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.97.62:45177)
0619 13:07:05.977675 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:05.977679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:05.977703 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:07:05.977883 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:05.977886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:05.977947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:05.978041 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:05.979233 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:05.980043 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:05.981224 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:05.981683 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:05.981781 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:05.985974 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:05.985979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:05.985982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:05.986017 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:05.988629 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:05.989205 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:05.989209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:05.989212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:05.989359 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:05.989819 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:05.989823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:05.989825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:05.990085 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:05.990288 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:05.990479 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 13:07:05.990735 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":292,"threads_started":1}
I20260619 13:07:05.994576 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:05.995186 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:06.001840 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 357552852383485186c459a617814527 (DEFAULT_TABLE table=test-table [id=d966763893ad461c98a2343121c34f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.001996 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 357552852383485186c459a617814527 (DEFAULT_TABLE table=test-table [id=d966763893ad461c98a2343121c34f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.002293 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357552852383485186c459a617814527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.002321 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 357552852383485186c459a617814527 (DEFAULT_TABLE table=test-table [id=d966763893ad461c98a2343121c34f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.002494 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357552852383485186c459a617814527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.002696 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357552852383485186c459a617814527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.007313 2740 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Bootstrap starting.
I20260619 13:07:06.008977 2741 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Bootstrap starting.
I20260619 13:07:06.009569 2740 tablet_bootstrap.cc:654] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.009748 2742 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Bootstrap starting.
I20260619 13:07:06.011237 2742 tablet_bootstrap.cc:654] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.011613 2741 tablet_bootstrap.cc:654] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.013697 2742 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: No bootstrap required, opened a new log
I20260619 13:07:06.013832 2742 ts_tablet_manager.cc:1397] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:07:06.013846 2741 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: No bootstrap required, opened a new log
I20260619 13:07:06.013981 2741 ts_tablet_manager.cc:1397] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 13:07:06.014477 2740 tablet_bootstrap.cc:492] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: No bootstrap required, opened a new log
I20260619 13:07:06.014617 2740 ts_tablet_manager.cc:1397] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:07:06.014608 2742 raft_consensus.cc:348] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.014868 2742 raft_consensus.cc:374] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.014729 2741 raft_consensus.cc:348] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.014961 2741 raft_consensus.cc:374] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.014963 2742 raft_consensus.cc:729] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.015051 2741 raft_consensus.cc:729] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.015202 2742 consensus_queue.cc:260] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.015304 2741 consensus_queue.cc:260] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.015393 2740 raft_consensus.cc:348] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.015618 2740 raft_consensus.cc:374] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.015718 2740 raft_consensus.cc:729] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.015969 2740 consensus_queue.cc:260] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.016477 2741 ts_tablet_manager.cc:1428] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:07:06.016485 2740 ts_tablet_manager.cc:1428] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 13:07:06.017139 2742 ts_tablet_manager.cc:1428] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:06.287084 2709 raft_consensus.cc:482] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:06.287374 2709 raft_consensus.cc:504] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.288164 2709 leader_election.cc:288] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:06.288786 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357552852383485186c459a617814527" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" is_pre_election: true
I20260619 13:07:06.288772 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357552852383485186c459a617814527" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" is_pre_election: true
I20260619 13:07:06.289048 2576 raft_consensus.cc:2454] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:06.289048 2299 raft_consensus.cc:2454] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 0.
I20260619 13:07:06.289584 2370 leader_election.cc:302] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:06.289930 2709 raft_consensus.cc:2781] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:06.290038 2709 raft_consensus.cc:482] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:06.290171 2709 raft_consensus.cc:3037] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.292663 2709 raft_consensus.cc:504] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.293273 2709 leader_election.cc:288] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Requested vote from peers c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:06.293905 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357552852383485186c459a617814527" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68"
I20260619 13:07:06.293867 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357552852383485186c459a617814527" candidate_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
I20260619 13:07:06.294108 2299 raft_consensus.cc:3037] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.294113 2576 raft_consensus.cc:3037] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.296100 2299 raft_consensus.cc:2454] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:06.296240 2576 raft_consensus.cc:2454] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d42d46e8f0a41b3af2dbbf0410499f8 in term 1.
I20260619 13:07:06.296598 2370 leader_election.cc:302] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d42d46e8f0a41b3af2dbbf0410499f8, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:06.296954 2709 raft_consensus.cc:2781] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:06.297104 2709 raft_consensus.cc:686] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Running, Role: LEADER
I20260619 13:07:06.297371 2709 consensus_queue.cc:237] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.300159 2140 catalog_manager.cc:5495] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:06.321519 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:06.316086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.0.97.2:35969)
0619 13:07:06.316663 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:06.316667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:06.316683 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:07:06.316730 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:06.316733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:06.316790 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:06.316894 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:06.317266 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:06.318146 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:06.319424 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:06.319861 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:06.320001 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:06.320386 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:06.320891 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:06.320990 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:06.321125 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 13:07:06.321196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":331,"threads_started":1}
I20260619 13:07:06.330440 2576 raft_consensus.cc:1261] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:06.330493 2299 raft_consensus.cc:1261] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Refusing update from remote peer 1d42d46e8f0a41b3af2dbbf0410499f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:06.331090 2716 consensus_queue.cc:1035] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:06.331386 2709 consensus_queue.cc:1035] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:06.376488 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:06.376736 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:06.379483 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Sending DeleteTablet for 3 replicas of tablet 357552852383485186c459a617814527
I20260619 13:07:06.380357 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 357552852383485186c459a617814527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260619 13:07:06.380479 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 357552852383485186c459a617814527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35448
I20260619 13:07:06.380470 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 357552852383485186c459a617814527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60684
I20260619 13:07:06.380653 2729 tablet_replica.cc:330] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: stopping tablet replica
I20260619 13:07:06.380729 2731 tablet_replica.cc:330] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: stopping tablet replica
I20260619 13:07:06.380856 2729 raft_consensus.cc:2229] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:06.380916 2731 raft_consensus.cc:2229] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:06.381072 2730 tablet_replica.cc:330] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: stopping tablet replica
I20260619 13:07:06.381078 2729 raft_consensus.cc:2258] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:06.381085 2731 raft_consensus.cc:2258] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:06.381289 2730 raft_consensus.cc:2229] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:06.381680 2730 raft_consensus.cc:2258] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:06.381721 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 13:07:06.382294 2731 ts_tablet_manager.cc:1905] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:06.382485 2729 ts_tablet_manager.cc:1905] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 13:07:06.382622 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:06.382623 2730 ts_tablet_manager.cc:1905] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:06.389304 2731 ts_tablet_manager.cc:1918] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:06.389474 2731 log.cc:1196] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/357552852383485186c459a617814527
I20260619 13:07:06.389768 2730 ts_tablet_manager.cc:1918] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:06.389921 2731 ts_tablet_manager.cc:1939] T 357552852383485186c459a617814527 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting consensus metadata
I20260619 13:07:06.389954 2730 log.cc:1196] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/357552852383485186c459a617814527
I20260619 13:07:06.390372 2730 ts_tablet_manager.cc:1939] T 357552852383485186c459a617814527 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting consensus metadata
I20260619 13:07:06.390558 2729 ts_tablet_manager.cc:1918] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:06.390602 2557 tablet_service.cc:1467] Processing CreateTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 (DEFAULT_TABLE table=test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.390776 2729 log.cc:1196] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/357552852383485186c459a617814527
I20260619 13:07:06.391064 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f501e0dce4ce0bbe906827d2b29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.391232 2729 ts_tablet_manager.cc:1939] T 357552852383485186c459a617814527 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting consensus metadata
I20260619 13:07:06.391633 2418 tablet_service.cc:1467] Processing CreateTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 (DEFAULT_TABLE table=test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.392055 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f501e0dce4ce0bbe906827d2b29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.392513 2126 catalog_manager.cc:4841] TS 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969): tablet 357552852383485186c459a617814527 (table test-table [id=d966763893ad461c98a2343121c34f8a]) successfully deleted
I20260619 13:07:06.392632 2278 tablet_service.cc:1467] Processing CreateTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 (DEFAULT_TABLE table=test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:06.392982 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f501e0dce4ce0bbe906827d2b29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:06.393623 2128 catalog_manager.cc:4841] TS 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475): tablet 357552852383485186c459a617814527 (table test-table [id=d966763893ad461c98a2343121c34f8a]) successfully deleted
I20260619 13:07:06.393869 2127 catalog_manager.cc:4841] TS c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115): tablet 357552852383485186c459a617814527 (table test-table [id=d966763893ad461c98a2343121c34f8a]) successfully deleted
I20260619 13:07:06.397396 2740 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Bootstrap starting.
I20260619 13:07:06.397753 2742 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Bootstrap starting.
I20260619 13:07:06.399268 2740 tablet_bootstrap.cc:654] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.399323 2742 tablet_bootstrap.cc:654] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.399478 2741 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Bootstrap starting.
I20260619 13:07:06.401302 2741 tablet_bootstrap.cc:654] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:06.401496 2742 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: No bootstrap required, opened a new log
I20260619 13:07:06.401693 2742 ts_tablet_manager.cc:1397] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 13:07:06.402463 2740 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: No bootstrap required, opened a new log
I20260619 13:07:06.402628 2740 ts_tablet_manager.cc:1397] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 13:07:06.402465 2742 raft_consensus.cc:348] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.402734 2742 raft_consensus.cc:374] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.402865 2742 raft_consensus.cc:729] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.403146 2742 consensus_queue.cc:260] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.403311 2741 tablet_bootstrap.cc:492] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: No bootstrap required, opened a new log
I20260619 13:07:06.403479 2741 ts_tablet_manager.cc:1397] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:07:06.403347 2740 raft_consensus.cc:348] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.403579 2740 raft_consensus.cc:374] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.403656 2740 raft_consensus.cc:729] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d42d46e8f0a41b3af2dbbf0410499f8, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.403789 2742 ts_tablet_manager.cc:1428] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:07:06.403872 2740 consensus_queue.cc:260] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.404235 2741 raft_consensus.cc:348] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.404479 2741 raft_consensus.cc:374] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:06.404539 2740 ts_tablet_manager.cc:1428] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:07:06.404560 2741 raft_consensus.cc:729] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Initialized, Role: FOLLOWER
I20260619 13:07:06.404814 2741 consensus_queue.cc:260] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.406100 2741 ts_tablet_manager.cc:1428] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 13:07:06.572866 2720 raft_consensus.cc:482] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:06.573122 2720 raft_consensus.cc:504] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.573894 2720 leader_election.cc:288] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:06.574481 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11f501e0dce4ce0bbe906827d2b29a8" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" is_pre_election: true
I20260619 13:07:06.574484 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11f501e0dce4ce0bbe906827d2b29a8" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" is_pre_election: true
I20260619 13:07:06.574718 2577 raft_consensus.cc:2454] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 0.
I20260619 13:07:06.574743 2437 raft_consensus.cc:2454] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 0.
I20260619 13:07:06.575212 2231 leader_election.cc:302] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:06.575477 2720 raft_consensus.cc:2781] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:06.575603 2720 raft_consensus.cc:482] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:06.575713 2720 raft_consensus.cc:3037] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.577848 2720 raft_consensus.cc:504] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.578400 2720 leader_election.cc:288] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 election: Requested vote from peers 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969), 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475)
I20260619 13:07:06.579032 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11f501e0dce4ce0bbe906827d2b29a8" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94"
I20260619 13:07:06.579032 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b11f501e0dce4ce0bbe906827d2b29a8" candidate_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8"
I20260619 13:07:06.579295 2577 raft_consensus.cc:3037] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.579300 2437 raft_consensus.cc:3037] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:06.582620 2437 raft_consensus.cc:2454] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 1.
I20260619 13:07:06.582621 2577 raft_consensus.cc:2454] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0b3e55ce941f0ae40c7cd03bc2e68 in term 1.
I20260619 13:07:06.583179 2231 leader_election.cc:302] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4f0a7cc1ed41ba91cd53c8a2d6fd94, c4d0b3e55ce941f0ae40c7cd03bc2e68; no voters:
I20260619 13:07:06.583557 2720 raft_consensus.cc:2781] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:06.583932 2720 raft_consensus.cc:686] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 LEADER]: Becoming Leader. State: Replica: c4d0b3e55ce941f0ae40c7cd03bc2e68, State: Running, Role: LEADER
I20260619 13:07:06.584244 2720 consensus_queue.cc:237] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } }
I20260619 13:07:06.586964 2139 catalog_manager.cc:5495] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0b3e55ce941f0ae40c7cd03bc2e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:06.608264 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:06.603039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.0.97.1:34115)
0619 13:07:06.603695 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:06.603700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:06.603790 (+ 90us) server_negotiation.cc:407] Connection header received
0619 13:07:06.604011 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:06.604015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:06.604075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:06.604158 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:06.604485 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:06.605200 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:06.606691 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:06.607068 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:06.607160 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:06.607593 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:06.607707 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:06.607776 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:06.607955 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 13:07:06.608013 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":430,"threads_started":1}
I20260619 13:07:06.617233 2577 raft_consensus.cc:1261] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Refusing update from remote peer c4d0b3e55ce941f0ae40c7cd03bc2e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:06.617512 2437 raft_consensus.cc:1261] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Refusing update from remote peer c4d0b3e55ce941f0ae40c7cd03bc2e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:06.617847 2757 consensus_queue.cc:1035] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4f0a7cc1ed41ba91cd53c8a2d6fd94" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:06.618189 2720 consensus_queue.cc:1035] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d42d46e8f0a41b3af2dbbf0410499f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:07:06.639320 2711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:07:08.227909 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:08.228211 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:08.232313 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff799081eb9044b4a02eb296152f45ee: Sending DeleteTablet for 3 replicas of tablet b11f501e0dce4ce0bbe906827d2b29a8
I20260619 13:07:08.233261 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260619 13:07:08.233326 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60684
I20260619 13:07:08.233467 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f501e0dce4ce0bbe906827d2b29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35448
I20260619 13:07:08.234318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2213
I20260619 13:07:08.246079 2785 tablet_replica.cc:330] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: stopping tablet replica
I20260619 13:07:08.246575 2785 raft_consensus.cc:2229] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:08.247411 2785 raft_consensus.cc:2258] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:08.248507 2786 tablet_replica.cc:330] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: stopping tablet replica
I20260619 13:07:08.248962 2786 raft_consensus.cc:2229] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:08.249305 2786 raft_consensus.cc:2258] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:08.249446 2785 ts_tablet_manager.cc:1905] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:08.250546 2786 ts_tablet_manager.cc:1905] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:08.251140 2787 tablet_replica.cc:330] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: stopping tablet replica
I20260619 13:07:08.251353 2787 raft_consensus.cc:2229] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:08.251592 2787 raft_consensus.cc:2258] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:08.252545 2787 ts_tablet_manager.cc:1905] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:08.258227 2786 ts_tablet_manager.cc:1918] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:08.258459 2786 log.cc:1196] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/b11f501e0dce4ce0bbe906827d2b29a8
I20260619 13:07:08.258522 2785 ts_tablet_manager.cc:1918] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:08.258730 2785 log.cc:1196] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/b11f501e0dce4ce0bbe906827d2b29a8
I20260619 13:07:08.259061 2786 ts_tablet_manager.cc:1939] T b11f501e0dce4ce0bbe906827d2b29a8 P 1d42d46e8f0a41b3af2dbbf0410499f8: Deleting consensus metadata
I20260619 13:07:08.259298 2785 ts_tablet_manager.cc:1939] T b11f501e0dce4ce0bbe906827d2b29a8 P c4d0b3e55ce941f0ae40c7cd03bc2e68: Deleting consensus metadata
I20260619 13:07:08.259791 2787 ts_tablet_manager.cc:1918] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:07:08.259984 2787 log.cc:1196] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/b11f501e0dce4ce0bbe906827d2b29a8
I20260619 13:07:08.260479 2787 ts_tablet_manager.cc:1939] T b11f501e0dce4ce0bbe906827d2b29a8 P 0f4f0a7cc1ed41ba91cd53c8a2d6fd94: Deleting consensus metadata
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:08.260545 2126 catalog_manager.cc:4841] TS 1d42d46e8f0a41b3af2dbbf0410499f8 (127.0.97.2:35969): tablet b11f501e0dce4ce0bbe906827d2b29a8 (table test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]) successfully deleted
I20260619 13:07:08.260596 2127 catalog_manager.cc:4841] TS c4d0b3e55ce941f0ae40c7cd03bc2e68 (127.0.97.1:34115): tablet b11f501e0dce4ce0bbe906827d2b29a8 (table test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]) successfully deleted
I20260619 13:07:08.261477 2128 catalog_manager.cc:4841] TS 0f4f0a7cc1ed41ba91cd53c8a2d6fd94 (127.0.97.3:34475): tablet b11f501e0dce4ce0bbe906827d2b29a8 (table test-table [id=31843eb2a24641f19f3f97a0fbc94c4b]) successfully deleted
I20260619 13:07:08.267419 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:08.236654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.0.97.1:34115)
0619 13:07:08.248863 (+ 12209us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:08.248870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:08.248896 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:07:08.248964 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:08.248972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:08.249053 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:08.249223 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:08.251258 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.252622 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.254796 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.255644 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.255787 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:08.261896 (+ 6109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:08.261912 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:08.261928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:08.261962 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:08.264623 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.265335 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.265343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.265345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.265497 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.265965 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.265969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.265971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.266393 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:08.266664 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:08.266945 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 13:07:08.267130 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12090,"thread_start_us":11952,"threads_started":1}
W20260619 13:07:08.268222 2344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35784
W20260619 13:07:08.268950 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:08.269182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2213
I20260619 13:07:08.292069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2353
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:08.307649 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:08.292976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.2:35969)
0619 13:07:08.293439 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:08.293443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:08.293458 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:07:08.293511 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:08.293514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:08.293580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:08.293749 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:08.294973 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.295723 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.296989 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.297386 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.297520 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:08.301700 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:08.301716 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:08.301734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:08.301774 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:08.304793 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.305477 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.305485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.305488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.305676 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.306069 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.306076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.306079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.306442 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:08.306745 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:08.307059 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 13:07:08.307308 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
W20260619 13:07:08.308362 2483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60720
W20260619 13:07:08.309278 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:08.309434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2353
I20260619 13:07:08.334430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2492
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:07:08.349797 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:08.335242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.97.3:34475)
0619 13:07:08.335814 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:08.335819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:08.335834 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:07:08.335887 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:08.335891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:08.335948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:08.336087 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:08.337489 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.338296 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.339532 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:08.339920 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:08.340083 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:08.344463 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:08.344493 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:08.344510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:08.344551 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:08.347260 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.347866 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.347874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.347876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.348034 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:08.348409 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:08.348412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:08.348414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:08.348728 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:08.348943 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:08.349211 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 13:07:08.349383 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":268,"threads_started":1}
W20260619 13:07:08.350450 2622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35452
W20260619 13:07:08.351090 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2492: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:08.351253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2492
I20260619 13:07:08.370419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2110
I20260619 13:07:08.371651 2182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:08.487275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2110
2026-06-19T13:07:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8421 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2795](info): set up 2 sockets
Jun 19 13:07:08 dist-test-slave-q11v krb5kdc[2795](info): commencing operation
krb5kdc: starting...
W20260619 13:07:09.610148 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:09 dist-test-slave-q11v krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:09.801211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37021
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:10.124917 2808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:10.125303 2808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:10.125476 2808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:10.133038 2808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:10.133224 2808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:10.133348 2808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:10.133453 2808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:10.133543 2808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:10.146291 2808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:10.148396 2808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:10.150774 2808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:10.167379 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:10.167402 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:10.167351 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:10.168776 2808 server_base.cc:1034] running on GCE node
I20260619 13:07:10.169654 2808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:10.171684 2808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:10.172920 2808 hybrid_clock.cc:648] HybridClock initialized: now 1781874430172859 us; error 51 us; skew 500 ppm
I20260619 13:07:10.173305 2808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:10 dist-test-slave-q11v krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:10.176524 2808 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:10.178074 2808 webserver.cc:460] Webserver started at http://127.0.97.62:35753/ using document root <none> and password file <none>
I20260619 13:07:10.178853 2808 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:10.178985 2808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:10.179334 2808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:10.182391 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "38a2f8f1c8e141d48b905d6b0cbf7271"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "02ea73aa6097645d045199844d552a70"
server_key_iv: "6ee3067b1add196d0e1f07d2d2818b1e"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.183190 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "38a2f8f1c8e141d48b905d6b0cbf7271"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "02ea73aa6097645d045199844d552a70"
server_key_iv: "6ee3067b1add196d0e1f07d2d2818b1e"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.188318 2808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:07:10.191847 2823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:10.193187 2808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:07:10.193372 2808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "38a2f8f1c8e141d48b905d6b0cbf7271"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "02ea73aa6097645d045199844d552a70"
server_key_iv: "6ee3067b1add196d0e1f07d2d2818b1e"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.193543 2808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:10.218335 2808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:10.222484 2808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:10.222774 2808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:10.242195 2808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37021
I20260619 13:07:10.242214 2874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37021 every 8 connection(s)
I20260619 13:07:10.243618 2808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:10.247813 2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:10.250248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2808
I20260619 13:07:10.250537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:10.251478 388 external_mini_cluster.cc:1428] Setting key 28c059804abd4e772e7bb3ae677f005a
I20260619 13:07:10.263078 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271: Bootstrap starting.
Jun 19 13:07:10 dist-test-slave-q11v krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:10.267146 2875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:10.268772 2875 log.cc:824] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:10.272532 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271: No bootstrap required, opened a new log
I20260619 13:07:10.275934 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:10.253943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.97.62:37021)
0619 13:07:10.254784 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:10.254800 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:10.254840 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:07:10.256373 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:10.256396 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:10.256808 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:10.257191 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:10.258694 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.260004 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.261308 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.262050 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.262238 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:10.266493 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:10.266505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:10.266525 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:10.266578 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:10.270943 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:10.271655 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:10.271665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:10.271673 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:10.271913 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:10.272279 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:10.272285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:10.272289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:10.273089 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:10.273446 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:10.273798 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 13:07:10.274272 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":384,"threads_started":1}
I20260619 13:07:10.283303 2875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260619 13:07:10.283679 2875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:10.283751 2875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a2f8f1c8e141d48b905d6b0cbf7271, State: Initialized, Role: FOLLOWER
I20260619 13:07:10.284291 2875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260619 13:07:10.284497 2875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:10.284592 2875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:10.284719 2875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:10.286864 2875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260619 13:07:10.287328 2875 leader_election.cc:302] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a2f8f1c8e141d48b905d6b0cbf7271; no voters:
I20260619 13:07:10.287911 2875 leader_election.cc:288] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:10.288064 2880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:10.288738 2880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [term 1 LEADER]: Becoming Leader. State: Replica: 38a2f8f1c8e141d48b905d6b0cbf7271, State: Running, Role: LEADER
I20260619 13:07:10.289705 2880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260619 13:07:10.290303 2875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:10.292850 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a2f8f1c8e141d48b905d6b0cbf7271. Latest consensus state: current_term: 1 leader_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } }
I20260619 13:07:10.292837 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2f8f1c8e141d48b905d6b0cbf7271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } }
I20260619 13:07:10.293277 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:10.293300 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:10.294811 2886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:10.299603 2886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:10.308176 2886 catalog_manager.cc:1347] Generated new cluster ID: 4fe8e78dc7df4bc0bf1658a8bee283ab
I20260619 13:07:10.308292 2886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:10.322791 2886 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:10.324074 2886 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:10.331243 2886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a2f8f1c8e141d48b905d6b0cbf7271: Generated new TSK 0
I20260619 13:07:10.332067 2886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:10.377748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:10.717409 2901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:10.717695 2901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:10.717872 2901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:10.725054 2901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:10.725168 2901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:10.725380 2901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:10.735749 2901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:10.737107 2901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:10.738516 2901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:10.751502 2907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:10.751593 2909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:10.751543 2906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:10.755164 2901 server_base.cc:1034] running on GCE node
I20260619 13:07:10.755834 2901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:10.756897 2901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:10.758140 2901 hybrid_clock.cc:648] HybridClock initialized: now 1781874430758090 us; error 50 us; skew 500 ppm
I20260619 13:07:10.758477 2901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:10 dist-test-slave-q11v krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:10.761921 2901 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:10.763523 2901 webserver.cc:460] Webserver started at http://127.0.97.1:37109/ using document root <none> and password file <none>
I20260619 13:07:10.764283 2901 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:10.764432 2901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:10.764758 2901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:10.767989 2901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "94e28c8a16fb4f90bc9ab16a03917c92"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "466e42567fcc8a8852a513ad10e68bbf"
server_key_iv: "6858a47e2be9729dbcde4efd63c2d0c1"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.768841 2901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "94e28c8a16fb4f90bc9ab16a03917c92"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "466e42567fcc8a8852a513ad10e68bbf"
server_key_iv: "6858a47e2be9729dbcde4efd63c2d0c1"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.774678 2901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:07:10.778782 2916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:10.780328 2901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:07:10.780469 2901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "94e28c8a16fb4f90bc9ab16a03917c92"
format_stamp: "Formatted at 2026-06-19 13:07:10 on dist-test-slave-q11v"
server_key: "466e42567fcc8a8852a513ad10e68bbf"
server_key_iv: "6858a47e2be9729dbcde4efd63c2d0c1"
server_key_version: "encryptionkey@0"
I20260619 13:07:10.780663 2901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:10.798671 2901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:10.803174 2901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:10.803421 2901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:10.804359 2901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:10.805724 2901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:10.805799 2901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:10.805892 2901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:10.805934 2901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:10.839885 2901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39669
I20260619 13:07:10.839943 3028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39669 every 8 connection(s)
I20260619 13:07:10.841248 2901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:10.848001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 2901
I20260619 13:07:10.848241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:10.849021 388 external_mini_cluster.cc:1428] Setting key 6c44687c55e6a0a2788f39873acca195
Jun 19 13:07:10 dist-test-slave-q11v krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:10.865141 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:10.843988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43185 (local address 127.0.97.62:37021)
0619 13:07:10.844471 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:10.844476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:10.845272 (+ 796us) server_negotiation.cc:407] Connection header received
0619 13:07:10.847159 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:10.847164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:10.847264 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:10.847393 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:10.849554 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.850445 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.852891 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.853279 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.853387 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:10.857869 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:10.857879 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:10.857886 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:10.857930 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:10.861509 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:10.862442 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:10.862451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:10.862457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:10.862614 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:10.863054 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:10.863061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:10.863068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:10.863446 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:10.863615 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:10.864398 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 13:07:10.864669 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":262,"threads_started":1}
I20260619 13:07:10.866071 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:10.844397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:43185)
0619 13:07:10.845108 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:10.845150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:10.846880 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:10.847655 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:10.847670 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:10.848472 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:10.849330 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:10.849356 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.850625 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.850630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:10.852651 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:10.852670 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:10.853516 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:10.853526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:10.853937 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:10.854720 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:10.854755 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:10.857567 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:10.861839 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:10.861848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:10.861866 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:10.862304 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:10.862725 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:10.862728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:10.862731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:10.862929 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:10.863617 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:10.863624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:10.863973 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 13:07:10.864231 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 13:07:10.864776 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":274,"threads_started":1}
I20260619 13:07:10.868006 3029 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260619 13:07:10.868387 3029 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:07:10.869189 3029 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:07:10.871531 2840 ts_manager.cc:194] Registered new tserver with Master: 94e28c8a16fb4f90bc9ab16a03917c92 (127.0.97.1:39669)
I20260619 13:07:10.873828 2840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:10.896804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:11.222958 3035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:11.223308 3035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:11.223496 3035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:11.230891 3035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:11.231083 3035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:11.231287 3035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:11.241111 3035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:11.242543 3035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:11.244017 3035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:11.257468 3043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:11.257378 3041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:11.257855 3040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:11.258486 3035 server_base.cc:1034] running on GCE node
I20260619 13:07:11.259117 3035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:11.260043 3035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:11.261257 3035 hybrid_clock.cc:648] HybridClock initialized: now 1781874431261213 us; error 48 us; skew 500 ppm
I20260619 13:07:11.261649 3035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:11 dist-test-slave-q11v krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:11.264801 3035 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:11.266301 3035 webserver.cc:460] Webserver started at http://127.0.97.2:43379/ using document root <none> and password file <none>
I20260619 13:07:11.266955 3035 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:11.267159 3035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:11.267458 3035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:11.270382 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "99d0078235fa4b41a6a9d2411b58a94f"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "3aa6f3ab327861cc3e435de1e2feafc8"
server_key_iv: "9e1f3ee07b43e9cd2b2ca04c9f7d6119"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.271097 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "99d0078235fa4b41a6a9d2411b58a94f"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "3aa6f3ab327861cc3e435de1e2feafc8"
server_key_iv: "9e1f3ee07b43e9cd2b2ca04c9f7d6119"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.276171 3035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:11.279798 3050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.281152 3035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:11.281342 3035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "99d0078235fa4b41a6a9d2411b58a94f"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "3aa6f3ab327861cc3e435de1e2feafc8"
server_key_iv: "9e1f3ee07b43e9cd2b2ca04c9f7d6119"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.281550 3035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:11.293951 3035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:11.298514 3035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:11.298807 3035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:11.299813 3035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:11.301127 3035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:11.301203 3035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.301355 3035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:11.301405 3035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.335247 3035 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41227
I20260619 13:07:11.335352 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41227 every 8 connection(s)
I20260619 13:07:11.336565 3035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:11.345278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3035
I20260619 13:07:11.345474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:11.345929 388 external_mini_cluster.cc:1428] Setting key 108cd98118524be6146977cbc8d485e2
Jun 19 13:07:11 dist-test-slave-q11v krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:11.357846 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:11.339326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54089 (local address 127.0.97.62:37021)
0619 13:07:11.339535 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:11.339541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:11.340741 (+ 1200us) server_negotiation.cc:407] Connection header received
0619 13:07:11.342561 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:11.342566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:11.342635 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:11.342728 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:11.344279 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.345051 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.347006 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.347470 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.347598 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:11.351326 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:11.351335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:11.351342 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:11.351387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:11.354436 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:11.355444 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:11.355448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:11.355450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:11.355580 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:11.356079 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:11.356082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:11.356084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:11.356330 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:11.356434 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:11.357361 (+ 927us) server_negotiation.cc:299] Negotiation successful
0619 13:07:11.357535 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:07:11.359493 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:11.339867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:54089)
0619 13:07:11.340565 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:11.340607 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:11.342300 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:11.342948 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:11.342956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:11.343460 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:11.344097 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:11.344116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.345206 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.345210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:11.346836 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:11.346856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.347706 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.347714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:11.347972 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:11.348667 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:11.348689 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:11.351096 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:11.354591 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:11.354604 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:11.354626 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:11.355316 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:11.355684 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:11.355692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:11.355697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:11.355968 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:11.356451 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:11.356463 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:11.356964 (+ 501us) client_negotiation.cc:769] Sending connection context
0619 13:07:11.357281 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 13:07:11.357932 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260619 13:07:11.361236 3163 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260619 13:07:11.361655 3163 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:11.362387 3163 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260619 13:07:11.363942 2840 ts_manager.cc:194] Registered new tserver with Master: 99d0078235fa4b41a6a9d2411b58a94f (127.0.97.2:41227)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:07:11.365180 2840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:11.391279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:11.714699 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:11.714986 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:11.715140 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:11.722548 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:11.722687 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:11.722873 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:11.733448 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:11.734998 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:11.737120 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:11.752682 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:11.752779 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:11.752755 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:11.753907 3168 server_base.cc:1034] running on GCE node
I20260619 13:07:11.754573 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:11.755548 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:11.756814 3168 hybrid_clock.cc:648] HybridClock initialized: now 1781874431756750 us; error 41 us; skew 500 ppm
I20260619 13:07:11.757162 3168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:11 dist-test-slave-q11v krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:11.760586 3168 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:11.762290 3168 webserver.cc:460] Webserver started at http://127.0.97.3:33247/ using document root <none> and password file <none>
I20260619 13:07:11.763024 3168 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:11.763171 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:11.763486 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:11.766506 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "4d987ecf2bd445cb89d3c5c9d461c78a"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "acd41587b3320fe63e22b7fcf661fdcb"
server_key_iv: "2daa16285abb3e2c18fbe53b8a357749"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.767315 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4d987ecf2bd445cb89d3c5c9d461c78a"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "acd41587b3320fe63e22b7fcf661fdcb"
server_key_iv: "2daa16285abb3e2c18fbe53b8a357749"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.772715 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:11.776253 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.777696 3168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:11.777882 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "4d987ecf2bd445cb89d3c5c9d461c78a"
format_stamp: "Formatted at 2026-06-19 13:07:11 on dist-test-slave-q11v"
server_key: "acd41587b3320fe63e22b7fcf661fdcb"
server_key_iv: "2daa16285abb3e2c18fbe53b8a357749"
server_key_version: "encryptionkey@0"
I20260619 13:07:11.778091 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:11.792528 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:11.796955 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:11.797251 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:11.798307 3168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:11.799741 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:11.799817 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.799974 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:11.800025 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:11.832273 3168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44363
I20260619 13:07:11.832301 3295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44363 every 8 connection(s)
I20260619 13:07:11.833566 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:11.840008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3168
I20260619 13:07:11.840216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:11.840783 388 external_mini_cluster.cc:1428] Setting key 86fe3fad991825cc14089dd6dc4bd7e1
Jun 19 13:07:11 dist-test-slave-q11v krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:11.853912 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:11.836151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39177 (local address 127.0.97.62:37021)
0619 13:07:11.836323 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:11.836327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:11.837565 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 13:07:11.839418 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:11.839424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:11.839517 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:11.839630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:11.841272 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.842408 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.844454 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.844853 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.844988 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:11.848493 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:11.848499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:11.848504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:11.848545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:11.850918 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:11.851631 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:11.851637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:11.851641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:11.851781 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:11.852196 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:11.852203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:11.852206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:11.852472 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:11.852627 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:11.853305 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 13:07:11.853526 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:07:11.855057 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:11.836673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:39177)
0619 13:07:11.837395 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:11.837441 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:11.839144 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:11.839867 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:11.839876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:11.840489 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:11.841107 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:11.841126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.842548 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.842554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:11.844305 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:11.844317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:11.845075 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:11.845083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:11.845377 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:11.846093 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:11.846116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:11.848260 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:11.851066 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:11.851075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:11.851092 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:11.851510 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:11.851900 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:11.851905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:11.851909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:11.852090 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:11.852603 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:11.852611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:11.852946 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 13:07:11.853208 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 13:07:11.853776 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
I20260619 13:07:11.856392 3296 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260619 13:07:11.856712 3296 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:11.857450 3296 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260619 13:07:11.858923 2839 ts_manager.cc:194] Registered new tserver with Master: 4d987ecf2bd445cb89d3c5c9d461c78a (127.0.97.3:44363)
I20260619 13:07:11.859786 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39177
I20260619 13:07:11.868969 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:07:11.877193 3029 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
W20260619 13:07:11.900972 3031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:11.898509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.97.62:37021)
0619 13:07:11.898704 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:11.898708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:11.899213 (+ 505us) server_negotiation.cc:407] Connection header received
0619 13:07:11.899344 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:11.899348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:11.899414 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:11.899508 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:11.900665 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34594: BlockingRecv error: recv got EOF from 127.0.0.1:34594 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:07:11.904126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2901
W20260619 13:07:11.906862 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2901: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39669: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:07:11.907060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2901
I20260619 13:07:11.922636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3035
W20260619 13:07:11.925118 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3035: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41227: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:07:11.925246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3035
I20260619 13:07:11.940235 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3168
W20260619 13:07:11.943024 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3168: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44363: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:07:11.943132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3168
I20260619 13:07:11.955140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 2808
I20260619 13:07:11.956324 2870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:12.058205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 2808
2026-06-19T13:07:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3567 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:12 dist-test-slave-q11v krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:12 dist-test-slave-q11v krb5kdc[3313](info): set up 2 sockets
Jun 19 13:07:12 dist-test-slave-q11v krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260619 13:07:13.145916 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:13 dist-test-slave-q11v krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:13.342165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37653
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:13.661772 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:13.662175 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:13.662343 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:13.669550 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:13.669734 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:13.669848 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:13.669956 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:13.670055 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:13.680203 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:13.681572 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:13.682940 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:13.696128 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:13.696080 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:13.696151 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:13.697036 3326 server_base.cc:1034] running on GCE node
I20260619 13:07:13.697975 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:13.699498 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:13.700704 3326 hybrid_clock.cc:648] HybridClock initialized: now 1781874433700664 us; error 34 us; skew 500 ppm
I20260619 13:07:13.701076 3326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:13 dist-test-slave-q11v krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:13.709036 3326 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:13.710718 3326 webserver.cc:460] Webserver started at http://127.0.97.62:40295/ using document root <none> and password file <none>
I20260619 13:07:13.711474 3326 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:13.711614 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:13.711941 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:13.715050 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "f266ab526b464f6bb6d0edd93466ef57"
format_stamp: "Formatted at 2026-06-19 13:07:13 on dist-test-slave-q11v"
server_key: "737e2b81cd8dc3bdac064ae1dfca9e44"
server_key_iv: "2722e3a5fff7405211c760498dc64150"
server_key_version: "encryptionkey@0"
I20260619 13:07:13.715862 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "f266ab526b464f6bb6d0edd93466ef57"
format_stamp: "Formatted at 2026-06-19 13:07:13 on dist-test-slave-q11v"
server_key: "737e2b81cd8dc3bdac064ae1dfca9e44"
server_key_iv: "2722e3a5fff7405211c760498dc64150"
server_key_version: "encryptionkey@0"
I20260619 13:07:13.721225 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260619 13:07:13.724859 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:13.726393 3326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:07:13.726575 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f266ab526b464f6bb6d0edd93466ef57"
format_stamp: "Formatted at 2026-06-19 13:07:13 on dist-test-slave-q11v"
server_key: "737e2b81cd8dc3bdac064ae1dfca9e44"
server_key_iv: "2722e3a5fff7405211c760498dc64150"
server_key_version: "encryptionkey@0"
I20260619 13:07:13.726796 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:13.752869 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:13.757368 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:13.757727 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:13.776372 3326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37653
I20260619 13:07:13.776407 3392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37653 every 8 connection(s)
I20260619 13:07:13.777842 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:13.780692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3326
I20260619 13:07:13.780894 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:13.781719 388 external_mini_cluster.cc:1428] Setting key 595401abe7a7e997862c60cbf5e0b46e
I20260619 13:07:13.782367 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:13.794761 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57: Bootstrap starting.
Jun 19 13:07:13 dist-test-slave-q11v krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:13.798573 3393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:13.799747 3393 log.cc:824] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:13.803192 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57: No bootstrap required, opened a new log
I20260619 13:07:13.805123 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:13.783702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.97.62:37653)
0619 13:07:13.784504 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:13.784517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:13.784556 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:13.786969 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:13.786992 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:13.787420 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:13.787866 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:13.789359 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:13.790704 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:13.791962 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:13.792688 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:13.792927 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:13.797306 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:13.797314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:13.797330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:13.797366 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:13.800927 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:13.801849 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:13.801856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:13.801863 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:13.802023 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:13.802438 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:13.802441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:13.802443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:13.802942 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:13.803142 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:13.803646 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 13:07:13.804053 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":369,"threads_started":1}
I20260619 13:07:13.809060 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:07:13.812502 3393 raft_consensus.cc:348] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } }
I20260619 13:07:13.812995 3393 raft_consensus.cc:374] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:13.813108 3393 raft_consensus.cc:729] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f266ab526b464f6bb6d0edd93466ef57, State: Initialized, Role: FOLLOWER
I20260619 13:07:13.813843 3393 consensus_queue.cc:260] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } }
I20260619 13:07:13.814100 3393 raft_consensus.cc:388] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:13.814242 3393 raft_consensus.cc:482] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:13.814411 3393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:13.816529 3393 raft_consensus.cc:504] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } }
I20260619 13:07:13.817059 3393 leader_election.cc:302] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f266ab526b464f6bb6d0edd93466ef57; no voters:
I20260619 13:07:13.817706 3393 leader_election.cc:288] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:13.817818 3399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:13.819761 3393 sys_catalog.cc:564] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:13.824712 3399 raft_consensus.cc:686] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [term 1 LEADER]: Becoming Leader. State: Replica: f266ab526b464f6bb6d0edd93466ef57, State: Running, Role: LEADER
I20260619 13:07:13.825400 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } }
I20260619 13:07:13.830982 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f266ab526b464f6bb6d0edd93466ef57. Latest consensus state: current_term: 1 leader_uuid: "f266ab526b464f6bb6d0edd93466ef57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } } }
I20260619 13:07:13.831393 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:13.831012 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f266ab526b464f6bb6d0edd93466ef57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f266ab526b464f6bb6d0edd93466ef57" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37653 } } }
I20260619 13:07:13.831528 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:13.834558 3413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:13.841408 3413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:13.852227 3413 catalog_manager.cc:1347] Generated new cluster ID: e3f4659271184f07859ad0f0bd9ef8f2
I20260619 13:07:13.852468 3413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:13.863270 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:13.854307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.0.97.62:37653)
0619 13:07:13.854478 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:13.854485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:13.855031 (+ 546us) server_negotiation.cc:407] Connection header received
0619 13:07:13.855199 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:13.855206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:13.855289 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:13.855403 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:13.856503 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:13.857681 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:13.859472 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:13.860149 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:13.860282 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:13.861492 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:13.861499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:13.861504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:13.862454 (+ 950us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:13.862511 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:13.862547 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:13.862655 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:13.862801 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 13:07:13.862901 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:07:13.877751 3413 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:13.879375 3413 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:13.891263 3413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f266ab526b464f6bb6d0edd93466ef57: Generated new TSK 0
I20260619 13:07:13.892331 3413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:07:13.905912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3326
I20260619 13:07:13.907677 3388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:13.999221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3326
2026-06-19T13:07:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1935 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:14 dist-test-slave-q11v krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:14 dist-test-slave-q11v krb5kdc[3427](info): set up 2 sockets
Jun 19 13:07:14 dist-test-slave-q11v krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260619 13:07:15.081910 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:15 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:15.272836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44389
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:15.595383 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:15.595687 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:15.595873 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:15.603156 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:15.603274 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:15.603358 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:15.603472 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:15.603554 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:15.613410 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:15.614765 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:15.616047 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:15.627653 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:15.627578 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:15.628189 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:15.628791 3440 server_base.cc:1034] running on GCE node
I20260619 13:07:15.629770 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:15.631265 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:15.632503 3440 hybrid_clock.cc:648] HybridClock initialized: now 1781874435632442 us; error 45 us; skew 500 ppm
I20260619 13:07:15.632964 3440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:15 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:15.636482 3440 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:15.638177 3440 webserver.cc:460] Webserver started at http://127.0.97.62:42383/ using document root <none> and password file <none>
I20260619 13:07:15.638921 3440 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:15.639062 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:15.639379 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:15.642469 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "37f6fabe3b434c7d84af56fae6b7c1da"
format_stamp: "Formatted at 2026-06-19 13:07:15 on dist-test-slave-q11v"
server_key: "f65f59fede3e515fc17eea0dd4fefc02"
server_key_iv: "acbc8a3cef617401546880d56ee1301c"
server_key_version: "encryptionkey@0"
I20260619 13:07:15.643183 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "37f6fabe3b434c7d84af56fae6b7c1da"
format_stamp: "Formatted at 2026-06-19 13:07:15 on dist-test-slave-q11v"
server_key: "f65f59fede3e515fc17eea0dd4fefc02"
server_key_iv: "acbc8a3cef617401546880d56ee1301c"
server_key_version: "encryptionkey@0"
I20260619 13:07:15.648499 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:15.652119 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:15.653565 3440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:15.653823 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "37f6fabe3b434c7d84af56fae6b7c1da"
format_stamp: "Formatted at 2026-06-19 13:07:15 on dist-test-slave-q11v"
server_key: "f65f59fede3e515fc17eea0dd4fefc02"
server_key_iv: "acbc8a3cef617401546880d56ee1301c"
server_key_version: "encryptionkey@0"
I20260619 13:07:15.654019 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:15.684314 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:15.688328 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:15.688629 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:15.706948 3440 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44389
I20260619 13:07:15.706977 3506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44389 every 8 connection(s)
I20260619 13:07:15.708387 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:15.711169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3440
I20260619 13:07:15.711381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:15.712165 388 external_mini_cluster.cc:1428] Setting key dc7573d4f4147b75eb54c027fed4d628
I20260619 13:07:15.712729 3507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:07:15 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:15.728170 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da: Bootstrap starting.
I20260619 13:07:15.731748 3507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:15.732880 3507 log.cc:824] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:15.736017 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da: No bootstrap required, opened a new log
I20260619 13:07:15.736063 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:15.714217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.97.62:44389)
0619 13:07:15.714992 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:15.715004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:15.715043 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:15.717073 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:15.717101 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:15.717487 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:15.717888 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:15.719904 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:15.721421 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:15.723341 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:15.724023 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:15.724170 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:15.728958 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:15.728966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:15.728980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:15.729019 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:15.732065 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:15.732738 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:15.732745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:15.732751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:15.732912 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:15.733310 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:15.733314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:15.733316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:15.733945 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:15.734269 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:15.734701 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 13:07:15.735106 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":364,"threads_started":1}
I20260619 13:07:15.745445 3507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } }
I20260619 13:07:15.745934 3507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:15.746052 3507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f6fabe3b434c7d84af56fae6b7c1da, State: Initialized, Role: FOLLOWER
I20260619 13:07:15.746600 3507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } }
I20260619 13:07:15.746825 3507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:15.746971 3507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:15.747169 3507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:15.749540 3507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } }
I20260619 13:07:15.750144 3507 leader_election.cc:302] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f6fabe3b434c7d84af56fae6b7c1da; no voters:
I20260619 13:07:15.750702 3507 leader_election.cc:288] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:15.750838 3512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:15.751549 3512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [term 1 LEADER]: Becoming Leader. State: Replica: 37f6fabe3b434c7d84af56fae6b7c1da, State: Running, Role: LEADER
I20260619 13:07:15.752106 3512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } }
I20260619 13:07:15.752851 3507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:15.754988 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f6fabe3b434c7d84af56fae6b7c1da. Latest consensus state: current_term: 1 leader_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } } }
I20260619 13:07:15.755367 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:15.756057 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f6fabe3b434c7d84af56fae6b7c1da" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44389 } } }
I20260619 13:07:15.756392 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:15.756434 3521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:15.763955 3521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:15.777066 3521 catalog_manager.cc:1347] Generated new cluster ID: 28dbaf0dbfd546fca68d985e7a7979dc
I20260619 13:07:15.777220 3521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:15.788938 3521 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:15.790895 3521 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:15.808878 3521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f6fabe3b434c7d84af56fae6b7c1da: Generated new TSK 0
I20260619 13:07:15.810132 3521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:15.865265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:16.198741 3533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:16.199036 3533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:16.199132 3533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:16.206408 3533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:16.206526 3533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:16.206640 3533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:16.216426 3533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:16.217758 3533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:16.219053 3533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:16.232421 3538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:16.232389 3539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:16.232366 3541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:16.233430 3533 server_base.cc:1034] running on GCE node
I20260619 13:07:16.234145 3533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:16.235212 3533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:16.236464 3533 hybrid_clock.cc:648] HybridClock initialized: now 1781874436236414 us; error 66 us; skew 500 ppm
I20260619 13:07:16.236799 3533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:16 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:16.240010 3533 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:16.241463 3533 webserver.cc:460] Webserver started at http://127.0.97.1:45583/ using document root <none> and password file <none>
I20260619 13:07:16.242194 3533 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:16.242322 3533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:16.242645 3533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:16.245569 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "00f849b814734476a2d824da81d67c98"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "0a85bd22c45ca0378e276561c04f3b19"
server_key_iv: "c60d28d95e7e36855260a2ba69e5c680"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.246657 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "00f849b814734476a2d824da81d67c98"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "0a85bd22c45ca0378e276561c04f3b19"
server_key_iv: "c60d28d95e7e36855260a2ba69e5c680"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.251945 3533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:07:16.255887 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.257535 3533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:07:16.257774 3533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "00f849b814734476a2d824da81d67c98"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "0a85bd22c45ca0378e276561c04f3b19"
server_key_iv: "c60d28d95e7e36855260a2ba69e5c680"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.257961 3533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:16.279261 3533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:16.283469 3533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:16.283730 3533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:16.284638 3533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:16.285980 3533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:16.286057 3533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.286140 3533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:16.286185 3533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.318563 3533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44227
I20260619 13:07:16.318595 3660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44227 every 8 connection(s)
I20260619 13:07:16.319826 3533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:16.324124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3533
I20260619 13:07:16.324333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:16.325165 388 external_mini_cluster.cc:1428] Setting key 20af9708ee768a1da40d4f4bea651133
Jun 19 13:07:16 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:16.343245 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:16.322422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36829 (local address 127.0.97.62:44389)
0619 13:07:16.322843 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:16.322847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:16.323982 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 13:07:16.326448 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:16.326452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:16.326515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:16.326613 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:16.328632 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.329331 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.331893 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.332276 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.332376 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:16.336161 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:16.336171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:16.336179 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:16.336225 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:16.339860 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:16.340597 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:16.340605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:16.340612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:16.340763 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:16.341243 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:16.341252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:16.341256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:16.341683 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:16.341852 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:16.342555 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 13:07:16.342824 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260619 13:07:16.343901 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:16.323015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44389 (local address 127.0.97.1:36829)
0619 13:07:16.323780 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:16.323834 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:16.326151 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:16.326896 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:16.326908 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:16.327609 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:16.328440 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:16.328470 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.329507 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.329515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:16.331743 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:16.331761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.332515 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.332525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:16.332834 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:16.333532 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:16.333560 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:16.335925 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:16.340017 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:16.340024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:16.340038 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:16.340452 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:16.340890 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:16.340895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:16.340899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:16.341118 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:16.341803 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:16.341809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:16.342149 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 13:07:16.342373 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 13:07:16.342767 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:07:16.345566 3661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44389
I20260619 13:07:16.345908 3661 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:16.346624 3661 heartbeater.cc:511] Master 127.0.97.62:44389 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:07:16.348703 3472 ts_manager.cc:194] Registered new tserver with Master: 00f849b814734476a2d824da81d67c98 (127.0.97.1:44227)
I20260619 13:07:16.350174 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:16.370898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:16.701058 3667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:16.701400 3667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:16.701573 3667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:16.710670 3667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:16.710795 3667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:16.710973 3667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:16.726517 3667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:16.728006 3667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:16.729650 3667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:16.743567 3673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:16.745116 3672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:16.746078 3675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:16.754698 3667 server_base.cc:1034] running on GCE node
I20260619 13:07:16.755673 3667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:16.756916 3667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:16.758229 3667 hybrid_clock.cc:648] HybridClock initialized: now 1781874436758080 us; error 105 us; skew 500 ppm
I20260619 13:07:16.758634 3667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:16 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:16.762213 3667 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:16.764055 3667 webserver.cc:460] Webserver started at http://127.0.97.2:36761/ using document root <none> and password file <none>
I20260619 13:07:16.764813 3667 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:16.764945 3667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:16.765273 3667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:16.768303 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "5e700bf22fe44b7caed3a15a1c986629"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "57d7250d802dc19841d92cd87aa09ebd"
server_key_iv: "25e7d45becdad74eef9c8b092a2e5dcd"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.769134 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5e700bf22fe44b7caed3a15a1c986629"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "57d7250d802dc19841d92cd87aa09ebd"
server_key_iv: "25e7d45becdad74eef9c8b092a2e5dcd"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.774600 3667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:07:16.778257 3682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.779780 3667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:16.779951 3667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "5e700bf22fe44b7caed3a15a1c986629"
format_stamp: "Formatted at 2026-06-19 13:07:16 on dist-test-slave-q11v"
server_key: "57d7250d802dc19841d92cd87aa09ebd"
server_key_iv: "25e7d45becdad74eef9c8b092a2e5dcd"
server_key_version: "encryptionkey@0"
I20260619 13:07:16.780138 3667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:16.794384 3667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:16.798708 3667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:16.799021 3667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:16.800057 3667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:16.801503 3667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:16.801576 3667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.801846 3667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:16.801954 3667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:16.840548 3667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43427
I20260619 13:07:16.840596 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43427 every 8 connection(s)
I20260619 13:07:16.841874 3667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:16.850021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3667
I20260619 13:07:16.850194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:16.850644 388 external_mini_cluster.cc:1428] Setting key 7dfd0f27aa07ebb26bf306f2508ab497
Jun 19 13:07:16 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:16.863309 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:16.844398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55215 (local address 127.0.97.62:44389)
0619 13:07:16.844810 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:16.844815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:16.845884 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 13:07:16.847707 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:16.847712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:16.847806 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:16.847925 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:16.849421 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.850265 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.852209 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.852874 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.853000 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:16.856839 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:16.856848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:16.856856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:16.856899 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:16.859841 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:16.860790 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:16.860794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:16.860796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:16.860946 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:16.861464 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:16.861467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:16.861469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:16.861830 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:16.861963 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:16.862861 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 13:07:16.863056 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":225,"threads_started":1}
I20260619 13:07:16.864915 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:16.845018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44389 (local address 127.0.97.2:55215)
0619 13:07:16.845730 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:16.845777 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:16.847405 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:16.848161 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:16.848174 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:16.848699 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:16.849254 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:16.849270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.850429 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.850434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:16.852054 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:16.852068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:16.853141 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:16.853148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:16.853343 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:16.854099 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:16.854120 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:16.856595 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:16.859999 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:16.860011 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:16.860032 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:16.860677 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:16.861091 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:16.861099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:16.861106 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:16.861353 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:16.861948 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:16.861959 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:16.862430 (+ 471us) client_negotiation.cc:769] Sending connection context
0619 13:07:16.862775 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 13:07:16.863392 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":259,"threads_started":1}
I20260619 13:07:16.866693 3795 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44389
I20260619 13:07:16.867062 3795 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:16.867918 3795 heartbeater.cc:511] Master 127.0.97.62:44389 requested a full tablet report, sending...
I20260619 13:07:16.869585 3472 ts_manager.cc:194] Registered new tserver with Master: 5e700bf22fe44b7caed3a15a1c986629 (127.0.97.2:43427)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:07:16.870828 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:16.895098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:17.214771 3801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:17.215122 3801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:17.215303 3801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:17.222424 3801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:17.222584 3801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:17.222781 3801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:17.232805 3801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:17.234301 3801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:17.235657 3801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:17.248878 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:17.248966 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:17.248987 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:17.249507 3801 server_base.cc:1034] running on GCE node
I20260619 13:07:17.250209 3801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:17.251140 3801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:17.252386 3801 hybrid_clock.cc:648] HybridClock initialized: now 1781874437252338 us; error 53 us; skew 500 ppm
I20260619 13:07:17.252758 3801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:17.256137 3801 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:17.257673 3801 webserver.cc:460] Webserver started at http://127.0.97.3:36337/ using document root <none> and password file <none>
I20260619 13:07:17.258534 3801 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:17.258677 3801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:17.258978 3801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:17.262077 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "fcfe01d337b0456589347df766aebb20"
format_stamp: "Formatted at 2026-06-19 13:07:17 on dist-test-slave-q11v"
server_key: "52bfd607d2ca0c99e03ce86f84840dfc"
server_key_iv: "aaec3db0a342d62aab9f876f04ac4df6"
server_key_version: "encryptionkey@0"
I20260619 13:07:17.262993 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fcfe01d337b0456589347df766aebb20"
format_stamp: "Formatted at 2026-06-19 13:07:17 on dist-test-slave-q11v"
server_key: "52bfd607d2ca0c99e03ce86f84840dfc"
server_key_iv: "aaec3db0a342d62aab9f876f04ac4df6"
server_key_version: "encryptionkey@0"
I20260619 13:07:17.268095 3801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:17.271438 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:17.272811 3801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:17.272991 3801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "fcfe01d337b0456589347df766aebb20"
format_stamp: "Formatted at 2026-06-19 13:07:17 on dist-test-slave-q11v"
server_key: "52bfd607d2ca0c99e03ce86f84840dfc"
server_key_iv: "aaec3db0a342d62aab9f876f04ac4df6"
server_key_version: "encryptionkey@0"
I20260619 13:07:17.273190 3801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:17.289214 3801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:17.293011 3801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:17.293318 3801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:17.294332 3801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:17.295624 3801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:17.295728 3801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:17.295873 3801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:17.295948 3801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:17.328395 3801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45471
I20260619 13:07:17.328428 3928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45471 every 8 connection(s)
I20260619 13:07:17.329677 3801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:17.333835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3801
I20260619 13:07:17.334022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:17.334455 388 external_mini_cluster.cc:1428] Setting key 7895fc2df8e026b3ca16c245aeae27d6
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:17.348125 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.332100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41643 (local address 127.0.97.62:44389)
0619 13:07:17.332279 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:17.332284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:17.333348 (+ 1064us) server_negotiation.cc:407] Connection header received
0619 13:07:17.335291 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:17.335296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:17.335397 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:17.335501 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:17.336917 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.337709 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.339201 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.339594 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.339722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.342871 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:17.342877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:17.342882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:17.342920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:17.345281 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.346030 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.346037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.346041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.346157 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.346563 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.346568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.346571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.346818 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:17.346963 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:17.347585 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 13:07:17.347800 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:07:17.349187 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.332517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44389 (local address 127.0.97.3:41643)
0619 13:07:17.333200 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:17.333235 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:17.335034 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:17.335732 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:17.335740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:17.336222 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:17.336756 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:17.336772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.337865 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.337869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:17.339061 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:17.339073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.339804 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.339811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:17.340033 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.340620 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:17.340641 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:17.342682 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:17.345419 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:17.345429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:17.345444 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:17.345917 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:17.346262 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:17.346268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:17.346271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:17.346451 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:17.346939 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:17.346946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:17.347245 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 13:07:17.347484 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 13:07:17.347982 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260619 13:07:17.350455 3929 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44389
I20260619 13:07:17.350736 3929 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:17.351430 3929 heartbeater.cc:511] Master 127.0.97.62:44389 requested a full tablet report, sending...
I20260619 13:07:17.352737 3472 ts_manager.cc:194] Registered new tserver with Master: fcfe01d337b0456589347df766aebb20 (127.0.97.3:45471)
I20260619 13:07:17.353559 3661 heartbeater.cc:503] Master 127.0.97.62:44389 was elected leader, sending a full tablet report...
I20260619 13:07:17.353826 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41643
I20260619 13:07:17.361238 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:17.408922 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.395525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.97.62:44389)
0619 13:07:17.395762 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:17.395770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:17.396054 (+ 284us) server_negotiation.cc:407] Connection header received
0619 13:07:17.396271 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:17.396275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:17.396338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:17.396445 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:17.397824 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.398571 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.399781 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.400189 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.400326 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.404025 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:17.404030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:17.404034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:17.404066 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:17.406391 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.406999 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.407006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.407009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.407127 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.407496 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.407502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.407505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.407808 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:17.408089 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:17.408336 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 13:07:17.408546 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:07:17.409461 3471 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37132
I20260619 13:07:17.410938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3533
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:17.426383 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.412112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.97.1:44227)
0619 13:07:17.412682 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:17.412698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:17.412721 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:07:17.412800 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:17.412806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:17.413019 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:17.413183 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:17.414466 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.415401 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.416585 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.417131 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.417264 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.420936 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:17.420946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:17.420960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:17.420990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:17.423622 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.424338 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.424345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.424348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.424477 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.424839 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.424845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.424849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.425187 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:17.425517 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:17.425772 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 13:07:17.426040 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":237,"threads_started":1}
W20260619 13:07:17.427727 3656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46182
W20260619 13:07:17.428929 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3533: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:17.429121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3533
I20260619 13:07:17.445075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3667
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:17.460845 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.446090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.0.97.2:43427)
0619 13:07:17.446565 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:17.446572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:17.446597 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:07:17.446652 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:17.446658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:17.446863 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:17.446991 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:17.448083 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.449041 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.450239 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.450759 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.450853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.454892 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:17.454903 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:17.454917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:17.454949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:17.458417 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.459059 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.459068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.459072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.459193 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.459496 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.459499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.459501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.459809 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:17.460028 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:17.460263 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 13:07:17.460462 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":248,"threads_started":1}
W20260619 13:07:17.462183 3790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45562
W20260619 13:07:17.463496 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:17.463657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3667
I20260619 13:07:17.484748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3801
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:07:17.507853 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:17.485898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.97.3:45471)
0619 13:07:17.486714 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:17.486724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:17.486762 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:07:17.486845 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:17.486851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:17.487069 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:17.487253 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:17.489216 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.491073 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.493031 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:17.493973 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:17.494111 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:17.500251 (+ 6140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:17.500264 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:17.500284 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:17.500325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:17.503849 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.504748 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.504758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.504761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.504907 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:17.505321 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:17.505326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:17.505329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:17.505776 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:17.506046 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:17.507198 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 13:07:17.507448 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
W20260619 13:07:17.509454 3924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38240
W20260619 13:07:17.510627 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:07:17.510824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3801
I20260619 13:07:17.527240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3440
I20260619 13:07:17.528460 3502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:17.629806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3440
2026-06-19T13:07:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3635 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3946](info): set up 2 sockets
Jun 19 13:07:17 dist-test-slave-q11v krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260619 13:07:18.714020 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:18 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:18.910048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39871
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:19.231431 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:19.231745 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:19.231907 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:19.239169 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:19.239288 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:19.239382 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:19.239488 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:19.239553 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:19.249346 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:19.250732 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:19.252063 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:19.264349 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:19.264309 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:19.264437 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:19.265121 3959 server_base.cc:1034] running on GCE node
I20260619 13:07:19.265942 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:19.267331 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:19.268529 3959 hybrid_clock.cc:648] HybridClock initialized: now 1781874439268472 us; error 59 us; skew 500 ppm
I20260619 13:07:19.268862 3959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:19 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:19.276645 3959 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:19.278360 3959 webserver.cc:460] Webserver started at http://127.0.97.62:39299/ using document root <none> and password file <none>
I20260619 13:07:19.279064 3959 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:19.279178 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:19.279438 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:19.282550 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "ff2d14a43908a0b0b3b0f80a7ac04826"
server_key_iv: "86090ea1e73743cfffaa04f39ea2628e"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.283300 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "ff2d14a43908a0b0b3b0f80a7ac04826"
server_key_iv: "86090ea1e73743cfffaa04f39ea2628e"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.288909 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:19.292796 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:19.294421 3959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:07:19.294587 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "ff2d14a43908a0b0b3b0f80a7ac04826"
server_key_iv: "86090ea1e73743cfffaa04f39ea2628e"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.294771 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:19.311573 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:19.315711 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:19.315946 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:19.334980 3959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39871
I20260619 13:07:19.335067 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39871 every 8 connection(s)
I20260619 13:07:19.336416 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:19.338505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 3959
I20260619 13:07:19.338734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:19.339378 388 external_mini_cluster.cc:1428] Setting key d5073e8e13228a9a999ad22050ea620c
I20260619 13:07:19.340646 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:07:19 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:19.354630 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24: Bootstrap starting.
I20260619 13:07:19.358554 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:19.359786 4026 log.cc:824] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:19.361689 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:19.341398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.0.97.62:39871)
0619 13:07:19.342215 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:19.342227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:19.342265 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:07:19.343840 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:19.343862 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:19.344265 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:19.344668 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:19.346207 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.347531 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.348760 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.349444 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.349686 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:19.353932 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:19.353940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:19.353955 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:19.353997 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:19.357074 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:19.357888 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:19.357895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:19.357902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:19.358063 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:19.358469 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:19.358473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:19.358475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:19.359064 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:19.359387 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:19.359817 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 13:07:19.360227 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":350,"threads_started":1}
I20260619 13:07:19.363626 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24: No bootstrap required, opened a new log
I20260619 13:07:19.372802 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } }
I20260619 13:07:19.373268 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:19.373402 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5321c91e0b4b45909a3b9a0e0f8e24, State: Initialized, Role: FOLLOWER
I20260619 13:07:19.374104 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } }
I20260619 13:07:19.374382 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:19.374557 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:19.374763 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:19.377228 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } }
I20260619 13:07:19.377919 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f5321c91e0b4b45909a3b9a0e0f8e24; no voters:
I20260619 13:07:19.378623 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:19.378748 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:19.380024 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5321c91e0b4b45909a3b9a0e0f8e24, State: Running, Role: LEADER
I20260619 13:07:19.380507 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } }
I20260619 13:07:19.381079 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:19.383494 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } } }
I20260619 13:07:19.386093 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:19.383483 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f5321c91e0b4b45909a3b9a0e0f8e24. Latest consensus state: current_term: 1 leader_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5321c91e0b4b45909a3b9a0e0f8e24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39871 } } }
I20260619 13:07:19.386833 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:19.388725 4037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:19.394877 4037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:19.404851 4037 catalog_manager.cc:1347] Generated new cluster ID: dedb58d3ec91439399d9468c156b009a
I20260619 13:07:19.404987 4037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:19.421806 4037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:19.423103 4037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:19.433099 4037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f5321c91e0b4b45909a3b9a0e0f8e24: Generated new TSK 0
I20260619 13:07:19.434036 4037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:19.481020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:19.808736 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:19.809020 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:19.809113 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:19.816447 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:19.816586 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:19.816702 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:19.826696 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:19.828033 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:19.829336 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:19.842821 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:19.842989 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:19.842780 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:19.844033 4052 server_base.cc:1034] running on GCE node
I20260619 13:07:19.844673 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:19.845670 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:19.846890 4052 hybrid_clock.cc:648] HybridClock initialized: now 1781874439846837 us; error 61 us; skew 500 ppm
I20260619 13:07:19.847203 4052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:19 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:19.850365 4052 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:19.851912 4052 webserver.cc:460] Webserver started at http://127.0.97.1:39837/ using document root <none> and password file <none>
I20260619 13:07:19.852605 4052 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:19.852751 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:19.853053 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:19.855939 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "401fcec886e941f386ac549382507281"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "1082e82c861a68a43f465b0a95447f95"
server_key_iv: "a0794a2c5442e3bcf6b6f9fdb50db3b2"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.856703 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "401fcec886e941f386ac549382507281"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "1082e82c861a68a43f465b0a95447f95"
server_key_iv: "a0794a2c5442e3bcf6b6f9fdb50db3b2"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.862020 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 13:07:19.865463 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:19.866834 4052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:19.867012 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "401fcec886e941f386ac549382507281"
format_stamp: "Formatted at 2026-06-19 13:07:19 on dist-test-slave-q11v"
server_key: "1082e82c861a68a43f465b0a95447f95"
server_key_iv: "a0794a2c5442e3bcf6b6f9fdb50db3b2"
server_key_version: "encryptionkey@0"
I20260619 13:07:19.867228 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:19.887234 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:19.891968 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:19.892266 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:19.893247 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:19.894666 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:19.894769 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:19.894917 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:19.894991 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:19.928615 4052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33327
I20260619 13:07:19.928752 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33327 every 8 connection(s)
I20260619 13:07:19.929991 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:19.940229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4052
I20260619 13:07:19.940467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:19.941183 388 external_mini_cluster.cc:1428] Setting key 3aa8c206ac30428e156c7120bf6e55bf
Jun 19 13:07:19 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:19.949443 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:19.932436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41627 (local address 127.0.97.62:39871)
0619 13:07:19.932931 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:19.932937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:19.933941 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 13:07:19.935797 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:19.935801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:19.935864 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:19.935960 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:19.937397 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.938153 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.939636 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.940240 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.940387 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:19.944096 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:19.944100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:19.944103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:19.944138 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:19.946567 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:19.947243 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:19.947252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:19.947257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:19.947376 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:19.947791 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:19.947798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:19.947802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:19.948055 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:19.948222 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:19.948873 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 13:07:19.949080 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":262,"threads_started":1}
I20260619 13:07:19.950462 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:19.933025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39871 (local address 127.0.97.1:41627)
0619 13:07:19.933787 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:19.933831 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:19.935549 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:19.936165 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:19.936173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:19.936705 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:19.937254 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:19.937270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.938297 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.938300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:19.939496 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:19.939507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:19.940481 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:19.940486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:19.940740 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:19.941450 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:19.941476 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:19.943885 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:19.946704 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:19.946713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:19.946729 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:19.947127 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:19.947491 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:19.947495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:19.947499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:19.947683 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:19.948204 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:19.948212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:19.948530 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 13:07:19.948798 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 13:07:19.949294 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":246,"threads_started":1}
I20260619 13:07:19.951973 4180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39871
I20260619 13:07:19.952273 4180 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:19.952957 4180 heartbeater.cc:511] Master 127.0.97.62:39871 requested a full tablet report, sending...
I20260619 13:07:19.955351 3991 ts_manager.cc:194] Registered new tserver with Master: 401fcec886e941f386ac549382507281 (127.0.97.1:33327)
I20260619 13:07:19.957623 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41627
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:19.983072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:20.314431 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:20.314756 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:20.314921 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:20.322156 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:20.322280 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:20.322468 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:20.332115 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:20.333380 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:20.334884 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:20.349653 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:20.349789 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:20.350133 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:20.351310 4186 server_base.cc:1034] running on GCE node
I20260619 13:07:20.351974 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:20.352921 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:20.354187 4186 hybrid_clock.cc:648] HybridClock initialized: now 1781874440354124 us; error 67 us; skew 500 ppm
I20260619 13:07:20.354547 4186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:20 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:20.357812 4186 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:20.359311 4186 webserver.cc:460] Webserver started at http://127.0.97.2:44827/ using document root <none> and password file <none>
I20260619 13:07:20.360021 4186 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:20.360159 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:20.360462 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:20.363538 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "d2fba25ffebd435f9965d432045b2a61"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a440d875a9cb5299f9e3ee9e39d39a78"
server_key_iv: "385d932e88d9aeca3577fe660c1db21b"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.364315 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d2fba25ffebd435f9965d432045b2a61"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a440d875a9cb5299f9e3ee9e39d39a78"
server_key_iv: "385d932e88d9aeca3577fe660c1db21b"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.369588 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:07:20.373721 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.375198 4186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:20.375391 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "d2fba25ffebd435f9965d432045b2a61"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a440d875a9cb5299f9e3ee9e39d39a78"
server_key_iv: "385d932e88d9aeca3577fe660c1db21b"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.375602 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:20.392740 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:20.397095 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:20.397405 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:20.398486 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:20.399900 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:20.399976 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.400139 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:20.400192 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.431654 4186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44575
I20260619 13:07:20.431769 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44575 every 8 connection(s)
I20260619 13:07:20.432955 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:20.442425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4186
I20260619 13:07:20.442622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:20.443171 388 external_mini_cluster.cc:1428] Setting key 8e6af25f83e178b3d3c9c4b413f9b052
Jun 19 13:07:20 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:20.453948 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:20.435655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41619 (local address 127.0.97.62:39871)
0619 13:07:20.435857 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:20.435863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:20.437155 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 13:07:20.438947 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:20.438952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:20.439027 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:20.439119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:20.440635 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.441330 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.443519 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.443925 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.444036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:20.447442 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:20.447447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:20.447451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:20.447483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:20.450355 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:20.451095 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:20.451103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:20.451109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:20.451264 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:20.451776 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:20.451784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:20.451791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:20.452155 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:20.452309 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:20.453000 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 13:07:20.453263 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:20.454432 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:20.436258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39871 (local address 127.0.97.2:41619)
0619 13:07:20.436975 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:20.437020 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:20.438700 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:20.439367 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:20.439376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:20.439922 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:20.440465 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:20.440485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.441453 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.441456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:20.443383 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:20.443396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.444146 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.444153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:20.444364 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:20.445085 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:20.445107 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:20.447223 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:20.450520 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:20.450527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:20.450540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:20.450959 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:20.451395 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:20.451400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:20.451405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:20.451657 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:20.452270 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:20.452276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:20.452623 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 13:07:20.452844 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 13:07:20.453236 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":296,"threads_started":1}
I20260619 13:07:20.455955 4314 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39871
I20260619 13:07:20.456314 4314 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:20.457309 4314 heartbeater.cc:511] Master 127.0.97.62:39871 requested a full tablet report, sending...
I20260619 13:07:20.459154 3991 ts_manager.cc:194] Registered new tserver with Master: d2fba25ffebd435f9965d432045b2a61 (127.0.97.2:44575)
I20260619 13:07:20.460503 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41619
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:20.488948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:20.819214 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:20.819514 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:20.819607 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:20.826778 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:20.826885 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:20.827034 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:20.836933 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:20.838272 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:20.839694 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:20.852757 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:20.852833 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:20.852793 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:20.853917 4319 server_base.cc:1034] running on GCE node
I20260619 13:07:20.854526 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:20.855479 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:20.856694 4319 hybrid_clock.cc:648] HybridClock initialized: now 1781874440856650 us; error 40 us; skew 500 ppm
I20260619 13:07:20.857013 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:20 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:20.860224 4319 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:20.861696 4319 webserver.cc:460] Webserver started at http://127.0.97.3:41575/ using document root <none> and password file <none>
I20260619 13:07:20.862370 4319 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:20.862469 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:20.862803 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:20.865680 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "fcc2d66508c246c481799249df85b54b"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a03161a1ff8ba1534d705cc5e14f346d"
server_key_iv: "d08f8190e9781a49864312ff67fe5a8a"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.866542 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fcc2d66508c246c481799249df85b54b"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a03161a1ff8ba1534d705cc5e14f346d"
server_key_iv: "d08f8190e9781a49864312ff67fe5a8a"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.872187 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:20.875788 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.877168 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:20.877333 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "fcc2d66508c246c481799249df85b54b"
format_stamp: "Formatted at 2026-06-19 13:07:20 on dist-test-slave-q11v"
server_key: "a03161a1ff8ba1534d705cc5e14f346d"
server_key_iv: "d08f8190e9781a49864312ff67fe5a8a"
server_key_version: "encryptionkey@0"
I20260619 13:07:20.877547 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:20.895814 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:20.899873 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:20.900156 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:20.901067 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:20.902418 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:20.902489 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.902657 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:20.902707 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:20.937167 4319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34179
I20260619 13:07:20.937202 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34179 every 8 connection(s)
I20260619 13:07:20.938448 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:20.948007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4319
I20260619 13:07:20.948202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:20.948719 388 external_mini_cluster.cc:1428] Setting key 8a1b4b8bd5a18b79675a76efcb651e47
Jun 19 13:07:20 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:20.957405 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:20.940914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46335 (local address 127.0.97.62:39871)
0619 13:07:20.941127 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:20.941134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:20.942255 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 13:07:20.944102 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:20.944106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:20.944168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:20.944264 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:20.945707 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.946482 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.947966 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.948600 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.948730 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:20.952340 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:20.952345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:20.952348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:20.952379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:20.954764 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:20.955440 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:20.955443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:20.955445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:20.955561 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:20.955911 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:20.955914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:20.955916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:20.956174 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:20.956282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:20.956983 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 13:07:20.957154 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:07:20.958420 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:20.941365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39871 (local address 127.0.97.3:46335)
0619 13:07:20.942108 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:20.942146 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:20.943853 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:20.944455 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:20.944463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:20.944955 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:20.945465 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:20.945481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.946613 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.946617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:20.947823 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:20.947834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:20.948817 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:20.948824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:20.949026 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:20.949811 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:20.949837 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:20.952158 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:20.954896 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:20.954903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:20.954926 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:20.955335 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:20.955655 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:20.955658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:20.955660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:20.955823 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:20.956305 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:20.956311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:20.956647 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 13:07:20.956887 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 13:07:20.957278 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":334,"threads_started":1}
I20260619 13:07:20.959827 4447 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39871
I20260619 13:07:20.960148 4447 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:20.960822 4447 heartbeater.cc:511] Master 127.0.97.62:39871 requested a full tablet report, sending...
I20260619 13:07:20.960808 4180 heartbeater.cc:503] Master 127.0.97.62:39871 was elected leader, sending a full tablet report...
I20260619 13:07:20.962456 3990 ts_manager.cc:194] Registered new tserver with Master: fcc2d66508c246c481799249df85b54b (127.0.97.3:34179)
I20260619 13:07:20.963440 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46335
I20260619 13:07:20.963892 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:07:20 dist-test-slave-q11v krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:21.012063 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:20.997163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.97.1:33327)
0619 13:07:20.997810 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:20.997817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:20.997843 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:07:20.997922 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:20.997927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:20.998171 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:20.998295 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:20.999740 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.000852 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.002169 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.002711 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.002808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:21.006683 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:21.006695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:21.006709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:21.006739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:21.009532 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.010198 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.010205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.010207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.010322 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.010669 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.010672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.010674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.010990 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:21.011216 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:21.011464 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 13:07:21.011652 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":290,"threads_started":1}
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:21.029485 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:21.016438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.97.62:39871)
0619 13:07:21.016617 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:21.016621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:21.016944 (+ 323us) server_negotiation.cc:407] Connection header received
0619 13:07:21.017085 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:21.017089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:21.017154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:21.017236 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:21.018534 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.019273 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.020506 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.020921 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.021026 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:21.024821 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:21.024826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:21.024830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:21.024861 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:21.027339 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.027915 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.027918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.027921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.028040 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.028371 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.028374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.028376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.028660 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:21.028838 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:21.029095 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 13:07:21.029261 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 13:07:21.030130 3990 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210
I20260619 13:07:21.031234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4052
I20260619 13:07:21.044293 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:21.032236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.0.97.1:33327)
0619 13:07:21.032455 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:21.032461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:21.032659 (+ 198us) server_negotiation.cc:407] Connection header received
0619 13:07:21.032828 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:21.032832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:21.032921 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:21.033005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:21.034480 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.035181 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.036391 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.036792 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.036891 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:21.039395 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:21.039403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:21.039407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:21.039444 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:21.042037 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.042690 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.042693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.042695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.042803 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.043125 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.043128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.043130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.043356 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:21.043521 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:21.043787 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 13:07:21.043941 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:07:21.044962 4175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:21.126432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4052
I20260619 13:07:21.142596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4186
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:21.159092 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:21.143549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.0.97.2:44575)
0619 13:07:21.144105 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:21.144112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:21.144138 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:07:21.144195 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:21.144201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:21.144416 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:21.144586 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:21.145791 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.146816 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.148035 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.148581 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.148712 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:21.152832 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:21.152842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:21.152857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:21.152901 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:21.155919 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.156694 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.156703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.156706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.156861 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.157290 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.157297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.157301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.157737 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:21.158073 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:21.158411 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 13:07:21.158696 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260619 13:07:21.160480 4309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:21.242213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4186
I20260619 13:07:21.258214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4319
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:07:21.275980 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:21.259349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.97.3:34179)
0619 13:07:21.260009 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:21.260016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:21.260046 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:07:21.260114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:21.260120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:21.260337 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:21.260548 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:21.262061 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.263060 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.264513 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:21.265082 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:21.265184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:21.269855 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:21.269873 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:21.269889 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:21.269933 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:21.273098 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.273985 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.273993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.273995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.274123 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:21.274489 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:21.274492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:21.274494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:21.274827 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:21.275076 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:21.275352 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 13:07:21.275572 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
I20260619 13:07:21.277215 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:21.362548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4319
I20260619 13:07:21.377926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 3959
I20260619 13:07:21.379184 4021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:21.481302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 3959
2026-06-19T13:07:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3852 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[4466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[4466](info): set up 2 sockets
Jun 19 13:07:21 dist-test-slave-q11v krb5kdc[4466](info): commencing operation
krb5kdc: starting...
W20260619 13:07:22.593997 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:22 dist-test-slave-q11v krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:22.792594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:23.111764 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:23.112108 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:23.112274 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:23.119294 4479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:23.119453 4479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:23.119560 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:23.119663 4479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:23.119753 4479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:23.129434 4479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:23.130772 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:23.132067 4479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:23.143613 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:23.143496 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:23.143563 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:23.144304 4479 server_base.cc:1034] running on GCE node
I20260619 13:07:23.145097 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:23.146548 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:23.147770 4479 hybrid_clock.cc:648] HybridClock initialized: now 1781874443147726 us; error 61 us; skew 500 ppm
I20260619 13:07:23.148123 4479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:23 dist-test-slave-q11v krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:23.151172 4479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:23.152300 4479 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:23.152439 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:23.152753 4479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:23.155712 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "279969bfc0004413a67cbf57e1c2c2c9"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "b48c344a6283da6f8cfe77cc0a8b6adb"
server_key_iv: "ab72ba734bbfd7b401e42b771ee18b6a"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.156509 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "279969bfc0004413a67cbf57e1c2c2c9"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "b48c344a6283da6f8cfe77cc0a8b6adb"
server_key_iv: "ab72ba734bbfd7b401e42b771ee18b6a"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.161407 4479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:07:23.164788 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:23.166167 4479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:23.166353 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "279969bfc0004413a67cbf57e1c2c2c9"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "b48c344a6283da6f8cfe77cc0a8b6adb"
server_key_iv: "ab72ba734bbfd7b401e42b771ee18b6a"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.166575 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:23.182242 4479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:23.186213 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:23.186492 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:23.204885 4479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46047
I20260619 13:07:23.204905 4544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46047 every 8 connection(s)
I20260619 13:07:23.205881 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:23.209473 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:23.211052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4479
I20260619 13:07:23.211288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:23.212105 388 external_mini_cluster.cc:1428] Setting key 9ea61e6048a9f045a6d45de620a140f1
I20260619 13:07:23.223887 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: Bootstrap starting.
Jun 19 13:07:23 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:23.227829 4545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:23.228968 4545 log.cc:824] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:23.231848 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: No bootstrap required, opened a new log
I20260619 13:07:23.235351 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:23.214211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.0.97.62:46047)
0619 13:07:23.215049 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:23.215061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:23.215100 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:23.216731 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:23.216758 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:23.217192 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:23.217519 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:23.219401 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.220781 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.222720 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.223463 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.223605 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:23.228212 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:23.228219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:23.228238 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:23.228279 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:23.231292 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:23.232000 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:23.232007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:23.232013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:23.232177 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:23.232581 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:23.232585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:23.232587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:23.233098 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:23.233410 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:23.233903 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 13:07:23.234323 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":393,"threads_started":1}
I20260619 13:07:23.240248 4545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } }
I20260619 13:07:23.240628 4545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:23.240731 4545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279969bfc0004413a67cbf57e1c2c2c9, State: Initialized, Role: FOLLOWER
I20260619 13:07:23.241258 4545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } }
I20260619 13:07:23.241496 4545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:23.241652 4545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:23.241820 4545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:23.243930 4545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } }
I20260619 13:07:23.244494 4545 leader_election.cc:302] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 279969bfc0004413a67cbf57e1c2c2c9; no voters:
I20260619 13:07:23.245106 4545 leader_election.cc:288] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:23.245244 4550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:23.245914 4550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 279969bfc0004413a67cbf57e1c2c2c9, State: Running, Role: LEADER
I20260619 13:07:23.246373 4550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } }
I20260619 13:07:23.247124 4545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:23.249150 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "279969bfc0004413a67cbf57e1c2c2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } } }
I20260619 13:07:23.250308 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:23.249620 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 279969bfc0004413a67cbf57e1c2c2c9. Latest consensus state: current_term: 1 leader_uuid: "279969bfc0004413a67cbf57e1c2c2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279969bfc0004413a67cbf57e1c2c2c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46047 } } }
I20260619 13:07:23.252785 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:23.254148 4557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:23.259270 4557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:23.268347 4557 catalog_manager.cc:1347] Generated new cluster ID: e48378921e0e4858a5ae4013bb45d528
I20260619 13:07:23.268474 4557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:23.277868 4557 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:23.279165 4557 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:23.285406 4557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: Generated new TSK 0
I20260619 13:07:23.286459 4557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:23.340865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:23.667776 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:23.668119 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:23.668269 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:23.675910 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:23.676024 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:23.676141 4571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:23.685891 4571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:23.687206 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:23.688544 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:23.701900 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:23.701929 4579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:23.701970 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:23.706969 4571 server_base.cc:1034] running on GCE node
I20260619 13:07:23.707630 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:23.708570 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:23.709794 4571 hybrid_clock.cc:648] HybridClock initialized: now 1781874443709753 us; error 55 us; skew 500 ppm
I20260619 13:07:23.710112 4571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:23 dist-test-slave-q11v krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:23.713208 4571 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:23.714490 4571 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:23.714634 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:23.714967 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:23.718315 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "4f0dd894093849919bf97e4b4b285390"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "382c8bb845a5a20baeeea1662844596b"
server_key_iv: "de0a02bf81e3af8b386084450548b366"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.719167 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4f0dd894093849919bf97e4b4b285390"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "382c8bb845a5a20baeeea1662844596b"
server_key_iv: "de0a02bf81e3af8b386084450548b366"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.724277 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:07:23.727753 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:23.729131 4571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:23.729346 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "4f0dd894093849919bf97e4b4b285390"
format_stamp: "Formatted at 2026-06-19 13:07:23 on dist-test-slave-q11v"
server_key: "382c8bb845a5a20baeeea1662844596b"
server_key_iv: "de0a02bf81e3af8b386084450548b366"
server_key_version: "encryptionkey@0"
I20260619 13:07:23.729588 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:23.747285 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:23.751731 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:23.752014 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:23.753063 4571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:23.754544 4571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:23.754668 4571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:23.754817 4571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:23.754905 4571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:23.786450 4571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39361
I20260619 13:07:23.786496 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39361 every 8 connection(s)
I20260619 13:07:23.787302 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:23.790482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4571
I20260619 13:07:23.790699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:23.791307 388 external_mini_cluster.cc:1428] Setting key 1206a1926f8f882184c48b4c026e7341
Jun 19 13:07:23 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:23.808048 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:23.789815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54201 (local address 127.0.97.62:46047)
0619 13:07:23.790303 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:23.790310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:23.791737 (+ 1427us) server_negotiation.cc:407] Connection header received
0619 13:07:23.793447 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:23.793452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:23.793531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:23.793687 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:23.795105 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.795819 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.797370 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.798110 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.798252 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:23.802162 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:23.802167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:23.802171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:23.802202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:23.804664 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:23.805672 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:23.805676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:23.805678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:23.805804 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:23.806300 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:23.806304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:23.806306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:23.806561 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:23.806660 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:23.807629 (+ 969us) server_negotiation.cc:299] Negotiation successful
0619 13:07:23.807801 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":195,"threads_started":1}
I20260619 13:07:23.809659 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:23.790467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46047 (local address 127.0.97.1:54201)
0619 13:07:23.791297 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:23.791343 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:23.793189 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:23.793903 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:23.793911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:23.794410 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:23.794944 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:23.794960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.795979 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.795983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:23.797212 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:23.797222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:23.798352 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:23.798358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:23.798553 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:23.799256 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:23.799279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:23.801945 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:23.804809 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:23.804821 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:23.804841 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:23.805512 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:23.805920 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:23.805928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:23.805935 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:23.806185 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:23.806715 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:23.806725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:23.807206 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 13:07:23.807545 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 13:07:23.808122 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:07:23.811178 4698 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46047
I20260619 13:07:23.811508 4698 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:23.812147 4698 heartbeater.cc:511] Master 127.0.97.62:46047 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:07:23.814534 4510 ts_manager.cc:194] Registered new tserver with Master: 4f0dd894093849919bf97e4b4b285390 (127.0.97.1:39361)
I20260619 13:07:23.816636 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:23.836159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:24.158875 4704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:24.159215 4704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:24.159392 4704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:24.166714 4704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:24.166880 4704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:24.167073 4704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:24.177227 4704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:24.178683 4704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:24.180015 4704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:24.192544 4712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:24.192657 4710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:24.192871 4709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:24.193627 4704 server_base.cc:1034] running on GCE node
I20260619 13:07:24.194182 4704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:24.195103 4704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:24.196336 4704 hybrid_clock.cc:648] HybridClock initialized: now 1781874444196288 us; error 40 us; skew 500 ppm
I20260619 13:07:24.196728 4704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:24 dist-test-slave-q11v krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:24.200021 4704 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:24.201174 4704 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:24.201269 4704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:24.201531 4704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:24.204430 4704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "b76c346e559740a1a9c3e0181e81c9cd"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "012b278eaf2090ecd734a089e2e1aed0"
server_key_iv: "042f3321a8469d67d378c5378f5256eb"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.205212 4704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b76c346e559740a1a9c3e0181e81c9cd"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "012b278eaf2090ecd734a089e2e1aed0"
server_key_iv: "042f3321a8469d67d378c5378f5256eb"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.210219 4704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:07:24.213685 4718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.215170 4704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:24.215353 4704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "b76c346e559740a1a9c3e0181e81c9cd"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "012b278eaf2090ecd734a089e2e1aed0"
server_key_iv: "042f3321a8469d67d378c5378f5256eb"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.215564 4704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:24.238620 4704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:24.242573 4704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:24.242851 4704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:24.243789 4704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:24.245102 4704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:24.245239 4704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.245365 4704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:24.245453 4704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.278805 4704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46823
I20260619 13:07:24.278895 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46823 every 8 connection(s)
I20260619 13:07:24.279702 4704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:24.285364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4704
I20260619 13:07:24.285570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:24.286027 388 external_mini_cluster.cc:1428] Setting key 2b010da4850abac6fd1e8aa3c8cb84fa
Jun 19 13:07:24 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:07:24.305088 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.282236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52881 (local address 127.0.97.62:46047)
0619 13:07:24.282412 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.282416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.283499 (+ 1083us) server_negotiation.cc:407] Connection header received
0619 13:07:24.285291 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.285297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.285392 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.285510 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:24.287307 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.288254 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.290844 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.291254 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.291372 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.299345 (+ 7973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:24.299350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:24.299353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:24.299385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:24.302048 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.302763 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.302768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.302778 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.302895 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.303314 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.303319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.303323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.303600 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:24.303755 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.304478 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.304705 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:24.306282 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.282673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46047 (local address 127.0.97.2:52881)
0619 13:07:24.283341 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:24.283378 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:24.285004 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:24.285861 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:24.285869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:24.286435 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:24.287118 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.287137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.288407 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.288412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.290676 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.290693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.291497 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.291506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.291819 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.296767 (+ 4948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:24.296807 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:24.299082 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:24.302190 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:24.302200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:24.302218 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:24.302631 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:24.303005 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:24.303011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:24.303015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:24.303194 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:24.303733 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:24.303742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:24.304080 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 13:07:24.304385 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 13:07:24.304941 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:07:24.307904 4831 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46047
I20260619 13:07:24.308259 4831 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:24.309079 4831 heartbeater.cc:511] Master 127.0.97.62:46047 requested a full tablet report, sending...
I20260619 13:07:24.310765 4510 ts_manager.cc:194] Registered new tserver with Master: b76c346e559740a1a9c3e0181e81c9cd (127.0.97.2:46823)
I20260619 13:07:24.311988 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:24.332525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:24.662302 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:24.662601 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:24.662706 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:24.670080 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:24.670295 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:24.670509 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:24.680817 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:24.682186 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:24.683530 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:24.697067 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:24.697141 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:24.697118 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:24.698689 4836 server_base.cc:1034] running on GCE node
I20260619 13:07:24.699306 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:24.700306 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:24.701527 4836 hybrid_clock.cc:648] HybridClock initialized: now 1781874444701480 us; error 43 us; skew 500 ppm
I20260619 13:07:24.701854 4836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:24 dist-test-slave-q11v krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:24.704958 4836 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:24.706101 4836 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:24.706194 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:24.706461 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:24.709270 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "7382d236b90f4c7e833553c375b52afe"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "4a9d00b8d1877ddeadce0234fda8449a"
server_key_iv: "b13364588b93cfef1637be70326ee64c"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.710067 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7382d236b90f4c7e833553c375b52afe"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "4a9d00b8d1877ddeadce0234fda8449a"
server_key_iv: "b13364588b93cfef1637be70326ee64c"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.715057 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:07:24.718456 4850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.719828 4836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:24.719990 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "7382d236b90f4c7e833553c375b52afe"
format_stamp: "Formatted at 2026-06-19 13:07:24 on dist-test-slave-q11v"
server_key: "4a9d00b8d1877ddeadce0234fda8449a"
server_key_iv: "b13364588b93cfef1637be70326ee64c"
server_key_version: "encryptionkey@0"
I20260619 13:07:24.720232 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:24.750015 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:24.754594 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:24.754823 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:24.755712 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:24.756987 4836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:24.757073 4836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.757143 4836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:24.757197 4836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:24.788798 4836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41979
I20260619 13:07:24.788820 4962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41979 every 8 connection(s)
I20260619 13:07:24.789800 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:24.792878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 4836
I20260619 13:07:24.793090 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:24.793710 388 external_mini_cluster.cc:1428] Setting key 60b72a92fbad57f487e4281ed7826eb0
Jun 19 13:07:24 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:24.809692 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.792398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37843 (local address 127.0.97.62:46047)
0619 13:07:24.792593 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.792599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.794022 (+ 1423us) server_negotiation.cc:407] Connection header received
0619 13:07:24.796378 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.796386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.796481 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.796555 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:24.798119 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.798856 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.800428 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.800800 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.800895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.804171 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:24.804175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:24.804179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:24.804214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:24.806532 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.807214 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.807219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.807222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.807381 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.807873 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.807878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.807880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.808182 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:24.808356 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.809235 (+ 879us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.809392 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:07:24.810747 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.792987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46047 (local address 127.0.97.3:37843)
0619 13:07:24.793852 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:24.793914 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:24.796067 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:24.796830 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:24.796838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:24.797320 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:24.797958 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.797976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.798981 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.798985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.800288 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.800305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.800979 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.800984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.801162 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.801862 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:24.801888 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:24.803968 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:24.806650 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:24.806657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:24.806670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:24.807079 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:24.807509 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:24.807514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:24.807517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:24.807734 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:24.808336 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:24.808342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:24.808771 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 13:07:24.809142 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 13:07:24.809556 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":325,"threads_started":1}
I20260619 13:07:24.812052 4963 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46047
I20260619 13:07:24.812350 4963 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:24.812978 4963 heartbeater.cc:511] Master 127.0.97.62:46047 requested a full tablet report, sending...
I20260619 13:07:24.814301 4510 ts_manager.cc:194] Registered new tserver with Master: 7382d236b90f4c7e833553c375b52afe (127.0.97.3:41979)
I20260619 13:07:24.815065 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37843
I20260619 13:07:24.819600 4698 heartbeater.cc:503] Master 127.0.97.62:46047 was elected leader, sending a full tablet report...
I20260619 13:07:24.821703 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:07:24.837821 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.826086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.97.62:46047)
0619 13:07:24.826279 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.826288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.826737 (+ 449us) server_negotiation.cc:407] Connection header received
0619 13:07:24.826869 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.826873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.826929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.827006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:24.828346 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.829020 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.830215 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.830616 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.830715 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.832983 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:24.832988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:24.832991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:24.833020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:24.835558 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.836144 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.836148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.836150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.836269 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:24.836570 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:24.836574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:24.836576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:24.836881 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:24.837057 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.837300 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.837444 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:07:24.841893 4509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:24.848717 4509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:24.869091 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.860245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39361 (local address 127.0.0.1:41472)
0619 13:07:24.861003 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:24.861040 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:24.861219 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:24.863023 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:24.863034 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:24.863086 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:24.863607 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.863623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.865189 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.865193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.866852 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.866864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.868118 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.868125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.868333 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.868400 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 13:07:24.868542 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 13:07:24.868715 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":1015,"threads_started":1}
I20260619 13:07:24.869500 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.860908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.97.1:39361)
0619 13:07:24.862454 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.862460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.862481 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:07:24.862589 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.862597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.862809 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.862956 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:24.863778 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.864887 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.867001 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.867918 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.868065 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.868298 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.868780 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.869021 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401,"thread_start_us":1244,"threads_started":1}
I20260619 13:07:24.873844 4633 tablet_service.cc:1467] Processing CreateTablet for tablet 717cb2c2cfe84233940dacb29ef7857e (DEFAULT_TABLE table=test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:24.875134 4633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717cb2c2cfe84233940dacb29ef7857e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:24.877375 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.861060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41979 (local address 127.0.0.1:39084)
0619 13:07:24.870170 (+ 9110us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:24.870188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:24.870393 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:24.871074 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:24.871079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:24.871102 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:24.871452 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.871466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.872955 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.872962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.874797 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.874811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.876520 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.876527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.876693 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.876737 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:07:24.876835 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:07:24.876968 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8972,"spinlock_wait_cycles":1920,"thread_start_us":8831,"threads_started":1}
I20260619 13:07:24.877386 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.861313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.97.3:41979)
0619 13:07:24.862138 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.862146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.870292 (+ 8146us) server_negotiation.cc:407] Connection header received
0619 13:07:24.870604 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.870615 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.870893 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.871100 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:24.871646 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.872834 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.874960 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.876327 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.876452 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.876665 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.876893 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.877045 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":483,"threads_started":1}
I20260619 13:07:24.878155 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.870253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46823 (local address 127.0.0.1:38894)
0619 13:07:24.870969 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:24.870988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:24.871161 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:24.871795 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:24.871800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:24.871824 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:24.872007 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.872021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.873753 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.873759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.875526 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:24.875536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.877383 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.877391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:24.877618 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.877651 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:07:24.877748 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:07:24.877918 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"spinlock_wait_cycles":896}
I20260619 13:07:24.878273 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:24.870457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.97.2:46823)
0619 13:07:24.870931 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:24.870942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:24.871298 (+ 356us) server_negotiation.cc:407] Connection header received
0619 13:07:24.871398 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:24.871409 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:24.871615 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:24.871802 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:24.872157 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.873588 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.875662 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:24.876988 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:24.877106 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:24.877285 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:24.877827 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 13:07:24.877979 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":245,"threads_started":1}
I20260619 13:07:24.881242 4766 tablet_service.cc:1467] Processing CreateTablet for tablet 717cb2c2cfe84233940dacb29ef7857e (DEFAULT_TABLE table=test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:24.881281 4898 tablet_service.cc:1467] Processing CreateTablet for tablet 717cb2c2cfe84233940dacb29ef7857e (DEFAULT_TABLE table=test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:24.882586 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717cb2c2cfe84233940dacb29ef7857e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:24.882647 4898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717cb2c2cfe84233940dacb29ef7857e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:24.891705 4980 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Bootstrap starting.
I20260619 13:07:24.894232 4981 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Bootstrap starting.
I20260619 13:07:24.895852 4980 tablet_bootstrap.cc:654] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:24.896874 4980 log.cc:824] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:24.898072 4981 tablet_bootstrap.cc:654] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:24.898213 4982 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Bootstrap starting.
I20260619 13:07:24.899231 4981 log.cc:824] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:24.899911 4980 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: No bootstrap required, opened a new log
I20260619 13:07:24.900157 4980 ts_tablet_manager.cc:1397] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 13:07:24.901753 4981 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: No bootstrap required, opened a new log
I20260619 13:07:24.901999 4981 ts_tablet_manager.cc:1397] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 13:07:24.904860 4982 tablet_bootstrap.cc:654] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:24.906075 4982 log.cc:824] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:24.909128 4982 tablet_bootstrap.cc:492] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: No bootstrap required, opened a new log
I20260619 13:07:24.909399 4982 ts_tablet_manager.cc:1397] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260619 13:07:24.909634 4981 raft_consensus.cc:348] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.909716 4980 raft_consensus.cc:348] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.910018 4981 raft_consensus.cc:374] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:24.910127 4981 raft_consensus.cc:729] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76c346e559740a1a9c3e0181e81c9cd, State: Initialized, Role: FOLLOWER
I20260619 13:07:24.910189 4980 raft_consensus.cc:374] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:24.910328 4980 raft_consensus.cc:729] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f0dd894093849919bf97e4b4b285390, State: Initialized, Role: FOLLOWER
I20260619 13:07:24.910811 4981 consensus_queue.cc:260] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.910954 4980 consensus_queue.cc:260] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.912698 4980 ts_tablet_manager.cc:1428] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260619 13:07:24.913694 4831 heartbeater.cc:503] Master 127.0.97.62:46047 was elected leader, sending a full tablet report...
I20260619 13:07:24.914253 4981 ts_tablet_manager.cc:1428] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260619 13:07:24.919744 4982 raft_consensus.cc:348] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.920104 4982 raft_consensus.cc:374] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:24.920207 4982 raft_consensus.cc:729] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7382d236b90f4c7e833553c375b52afe, State: Initialized, Role: FOLLOWER
I20260619 13:07:24.920717 4982 consensus_queue.cc:260] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:24.921686 4963 heartbeater.cc:503] Master 127.0.97.62:46047 was elected leader, sending a full tablet report...
I20260619 13:07:24.922286 4982 ts_tablet_manager.cc:1428] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260619 13:07:25.031648 4832 tablet.cc:2369] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:25.039471 4699 tablet.cc:2369] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:25.041249 4964 tablet.cc:2369] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:25.122840 4986 raft_consensus.cc:482] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:25.123238 4986 raft_consensus.cc:504] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:25.125137 4986 leader_election.cc:288] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f0dd894093849919bf97e4b4b285390 (127.0.97.1:39361), 7382d236b90f4c7e833553c375b52afe (127.0.97.3:41979)
I20260619 13:07:25.132009 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.125631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40949 (local address 127.0.97.1:39361)
0619 13:07:25.125848 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.125856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.126739 (+ 883us) server_negotiation.cc:407] Connection header received
0619 13:07:25.126880 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.126884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.126975 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.127068 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:25.127740 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.128488 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.130283 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.131008 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.131142 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.131259 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.131605 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.131741 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:07:25.132004 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.125755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45043 (local address 127.0.97.3:41979)
0619 13:07:25.125991 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.126000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.126594 (+ 594us) server_negotiation.cc:407] Connection header received
0619 13:07:25.126777 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.126784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.126869 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.126945 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:25.127599 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.128367 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.130275 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.131025 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.131179 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.131288 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.131605 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.131734 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:07:25.132176 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.125615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41979 (local address 127.0.97.2:45043)
0619 13:07:25.126470 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:25.126486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:25.126624 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:25.126986 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:25.126991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:25.127019 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:25.127449 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:25.127458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.128504 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.128509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:25.130124 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:25.130135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.131273 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.131282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:25.131428 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.131469 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 13:07:25.131578 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:07:25.131732 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":18,"thread_start_us":475,"threads_started":1}
I20260619 13:07:25.132226 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.125609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39361 (local address 127.0.97.2:40949)
0619 13:07:25.126587 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:25.126606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:25.126758 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:25.127099 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:25.127104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:25.127132 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:25.127560 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:25.127572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.128630 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.128633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:25.130126 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:25.130137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.131239 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.131245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:25.131431 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.131471 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 13:07:25.131577 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:07:25.131743 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":65,"thread_start_us":581,"threads_started":1}
I20260619 13:07:25.133375 4653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717cb2c2cfe84233940dacb29ef7857e" candidate_uuid: "b76c346e559740a1a9c3e0181e81c9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f0dd894093849919bf97e4b4b285390" is_pre_election: true
I20260619 13:07:25.134052 4653 raft_consensus.cc:2454] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76c346e559740a1a9c3e0181e81c9cd in term 0.
I20260619 13:07:25.134019 4918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717cb2c2cfe84233940dacb29ef7857e" candidate_uuid: "b76c346e559740a1a9c3e0181e81c9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7382d236b90f4c7e833553c375b52afe" is_pre_election: true
I20260619 13:07:25.134567 4918 raft_consensus.cc:2454] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76c346e559740a1a9c3e0181e81c9cd in term 0.
I20260619 13:07:25.134795 4721 leader_election.cc:302] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f0dd894093849919bf97e4b4b285390, b76c346e559740a1a9c3e0181e81c9cd; no voters:
I20260619 13:07:25.135178 4986 raft_consensus.cc:2781] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:25.135303 4986 raft_consensus.cc:482] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:25.135371 4986 raft_consensus.cc:3037] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:25.137473 4986 raft_consensus.cc:504] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:25.138058 4986 leader_election.cc:288] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [CANDIDATE]: Term 1 election: Requested vote from peers 4f0dd894093849919bf97e4b4b285390 (127.0.97.1:39361), 7382d236b90f4c7e833553c375b52afe (127.0.97.3:41979)
I20260619 13:07:25.138537 4653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717cb2c2cfe84233940dacb29ef7857e" candidate_uuid: "b76c346e559740a1a9c3e0181e81c9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f0dd894093849919bf97e4b4b285390"
I20260619 13:07:25.138537 4918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717cb2c2cfe84233940dacb29ef7857e" candidate_uuid: "b76c346e559740a1a9c3e0181e81c9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7382d236b90f4c7e833553c375b52afe"
I20260619 13:07:25.138741 4653 raft_consensus.cc:3037] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:25.138737 4918 raft_consensus.cc:3037] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:25.141829 4653 raft_consensus.cc:2454] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76c346e559740a1a9c3e0181e81c9cd in term 1.
I20260619 13:07:25.141829 4918 raft_consensus.cc:2454] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76c346e559740a1a9c3e0181e81c9cd in term 1.
I20260619 13:07:25.142179 4721 leader_election.cc:302] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f0dd894093849919bf97e4b4b285390, b76c346e559740a1a9c3e0181e81c9cd; no voters:
I20260619 13:07:25.142446 4986 raft_consensus.cc:2781] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:25.142879 4986 raft_consensus.cc:686] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 1 LEADER]: Becoming Leader. State: Replica: b76c346e559740a1a9c3e0181e81c9cd, State: Running, Role: LEADER
I20260619 13:07:25.143347 4986 consensus_queue.cc:237] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } }
I20260619 13:07:25.147864 4510 catalog_manager.cc:5495] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd reported cstate change: term changed from 0 to 1, leader changed from <none> to b76c346e559740a1a9c3e0181e81c9cd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b76c346e559740a1a9c3e0181e81c9cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76c346e559740a1a9c3e0181e81c9cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46823 } health_report { overall_health: HEALTHY } } }
I20260619 13:07:25.171741 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.166497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.97.2:46823)
0619 13:07:25.166716 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.166724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.166745 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:07:25.166982 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.166986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.167069 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.167224 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:25.167594 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.168281 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.169591 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.170034 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.170173 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.170639 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:25.171159 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:25.171321 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.171378 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.171455 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:07:25.180119 4918 raft_consensus.cc:1261] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 1 FOLLOWER]: Refusing update from remote peer b76c346e559740a1a9c3e0181e81c9cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:25.180117 4653 raft_consensus.cc:1261] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 1 FOLLOWER]: Refusing update from remote peer b76c346e559740a1a9c3e0181e81c9cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:25.181200 4991 consensus_queue.cc:1035] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f0dd894093849919bf97e4b4b285390" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:25.181538 4986 consensus_queue.cc:1035] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7382d236b90f4c7e833553c375b52afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:25.206720 4510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:25.207005 4510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:25.210261 4510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 279969bfc0004413a67cbf57e1c2c2c9: Sending DeleteTablet for 3 replicas of tablet 717cb2c2cfe84233940dacb29ef7857e
I20260619 13:07:25.211546 4633 tablet_service.cc:1514] Processing DeleteTablet for tablet 717cb2c2cfe84233940dacb29ef7857e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260619 13:07:25.211772 4898 tablet_service.cc:1514] Processing DeleteTablet for tablet 717cb2c2cfe84233940dacb29ef7857e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39084
I20260619 13:07:25.212734 4766 tablet_service.cc:1514] Processing DeleteTablet for tablet 717cb2c2cfe84233940dacb29ef7857e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38894
I20260619 13:07:25.218566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4571
I20260619 13:07:25.222118 5005 tablet_replica.cc:330] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: stopping tablet replica
I20260619 13:07:25.223577 5005 raft_consensus.cc:2229] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:25.224704 5005 raft_consensus.cc:2258] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:25.228493 5005 ts_tablet_manager.cc:1905] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:25.239193 5005 ts_tablet_manager.cc:1918] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:25.239516 5005 log.cc:1196] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/717cb2c2cfe84233940dacb29ef7857e
I20260619 13:07:25.240154 5005 ts_tablet_manager.cc:1939] T 717cb2c2cfe84233940dacb29ef7857e P 7382d236b90f4c7e833553c375b52afe: Deleting consensus metadata
I20260619 13:07:25.241811 5006 tablet_replica.cc:330] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: stopping tablet replica
I20260619 13:07:25.242235 4497 catalog_manager.cc:4841] TS 7382d236b90f4c7e833553c375b52afe (127.0.97.3:41979): tablet 717cb2c2cfe84233940dacb29ef7857e (table test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]) successfully deleted
I20260619 13:07:25.243249 5006 raft_consensus.cc:2229] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:25.244031 5006 raft_consensus.cc:2258] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:25.246546 5006 ts_tablet_manager.cc:1905] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:25.247131 5004 tablet_replica.cc:330] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: stopping tablet replica
I20260619 13:07:25.247557 5004 raft_consensus.cc:2229] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:25.248183 5004 raft_consensus.cc:2258] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:25.250381 5004 ts_tablet_manager.cc:1905] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:07:25 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:25.254575 5006 ts_tablet_manager.cc:1918] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:25.254807 5006 log.cc:1196] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/717cb2c2cfe84233940dacb29ef7857e
I20260619 13:07:25.255460 5006 ts_tablet_manager.cc:1939] T 717cb2c2cfe84233940dacb29ef7857e P b76c346e559740a1a9c3e0181e81c9cd: Deleting consensus metadata
I20260619 13:07:25.257252 4497 catalog_manager.cc:4841] TS b76c346e559740a1a9c3e0181e81c9cd (127.0.97.2:46823): tablet 717cb2c2cfe84233940dacb29ef7857e (table test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]) successfully deleted
I20260619 13:07:25.258112 5004 ts_tablet_manager.cc:1918] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:25.258301 5004 log.cc:1196] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/717cb2c2cfe84233940dacb29ef7857e
I20260619 13:07:25.258818 5004 ts_tablet_manager.cc:1939] T 717cb2c2cfe84233940dacb29ef7857e P 4f0dd894093849919bf97e4b4b285390: Deleting consensus metadata
I20260619 13:07:25.260702 4496 catalog_manager.cc:4841] TS 4f0dd894093849919bf97e4b4b285390 (127.0.97.1:39361): tablet 717cb2c2cfe84233940dacb29ef7857e (table test-table [id=6693ff9bb66f402eb0fb3c0af2c23cab]) successfully deleted
I20260619 13:07:25.261439 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.220320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.97.1:39361)
0619 13:07:25.220544 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.220553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.241029 (+ 20476us) server_negotiation.cc:407] Connection header received
0619 13:07:25.242809 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.242815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.242922 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.243015 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:25.244825 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.246009 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.247829 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.248518 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.248623 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.253909 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:25.253920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:25.253935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:25.253976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:25.257942 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.258879 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.258889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.258900 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.259076 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.259534 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.259539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.259542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.259974 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:25.260255 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.260847 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.261053 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:07:25.262085 4693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:25.386180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4571
I20260619 13:07:25.403649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4704
Jun 19 13:07:25 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:25.421820 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.404426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.97.2:46823)
0619 13:07:25.404603 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.404611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.404652 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:07:25.404864 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.404868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.404936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.405023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:25.406629 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.407433 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.409206 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.409834 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.409931 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.414956 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:25.414976 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:25.415004 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:25.415050 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:25.418902 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.419552 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.419563 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.419567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.419750 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.420100 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.420105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.420109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.420591 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:25.420944 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.421153 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.421373 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:07:25.422565 4826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:25.535445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4704
I20260619 13:07:25.554363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4836
Jun 19 13:07:25 dist-test-slave-q11v krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:07:25.568439 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:25.555173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.97.3:41979)
0619 13:07:25.555333 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:25.555338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:25.555355 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:25.555500 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:25.555503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:25.555576 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:25.555673 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:25.556859 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.557547 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.558806 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:25.559202 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:25.559295 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:25.563351 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:25.563366 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:25.563383 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:25.563422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:25.566004 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.566674 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.566682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.566684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.566844 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:25.567194 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:25.567197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:25.567199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:25.567549 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:25.567764 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:25.568020 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:07:25.568187 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:07:25.569249 4958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:25.656853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4836
I20260619 13:07:25.672968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 4479
I20260619 13:07:25.674098 4540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:25.782191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 4479
2026-06-19T13:07:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4299 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T13:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:25Z Disabled control of system clock
I20260619 13:07:25.847851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33191
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:07:26.176199 5015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:26.176489 5015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:26.176576 5015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:26.184017 5015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:26.184139 5015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:26.184176 5015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:26.184234 5015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:26.184278 5015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:26.193403 5015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:26.194701 5015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:26.196395 5015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:26.207902 5020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:26.207840 5023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:26.207937 5021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:26.208823 5015 server_base.cc:1034] running on GCE node
I20260619 13:07:26.209549 5015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:26.210942 5015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:26.212152 5015 hybrid_clock.cc:648] HybridClock initialized: now 1781874446212105 us; error 39 us; skew 500 ppm
I20260619 13:07:26.212452 5015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:26.214689 5015 webserver.cc:460] Webserver started at http://127.0.97.62:37865/ using document root <none> and password file <none>
I20260619 13:07:26.215402 5015 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:26.215489 5015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:26.215763 5015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:26.218647 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bcd2df781014cd8f1dc38f479caa1326"
server_key_iv: "6c42f362cdae8eb9414b8928a0766572"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.219379 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bcd2df781014cd8f1dc38f479caa1326"
server_key_iv: "6c42f362cdae8eb9414b8928a0766572"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.224350 5015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 13:07:26.227742 5029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:26.229061 5015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:26.229236 5015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bcd2df781014cd8f1dc38f479caa1326"
server_key_iv: "6c42f362cdae8eb9414b8928a0766572"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.229449 5015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:26.235118 5015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:26.235886 5015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:26.236186 5015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:26.254169 5015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33191
I20260619 13:07:26.254189 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33191 every 8 connection(s)
I20260619 13:07:26.255511 5015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:26.256064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5015
I20260619 13:07:26.256254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:26.257155 388 external_mini_cluster.cc:1428] Setting key 96f8f5523a3ee7a537e9a56db680390c
I20260619 13:07:26.260382 5081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:26.265475 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:26.259285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.0.97.62:33191)
0619 13:07:26.260246 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:26.260257 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:26.260296 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:26.261917 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:26.261938 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:26.262341 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:26.262737 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:26.263818 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:26.263824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:26.263836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:26.263848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:26.263934 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:26.263949 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:26.264185 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:26.264327 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 13:07:26.264580 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":334,"threads_started":1}
I20260619 13:07:26.271582 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: Bootstrap starting.
I20260619 13:07:26.275027 5081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:26.276049 5081 log.cc:824] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:26.279093 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: No bootstrap required, opened a new log
I20260619 13:07:26.286582 5081 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } }
I20260619 13:07:26.286978 5081 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:26.287078 5081 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b9ae7f9f9e4432a86a6c079738ab9e, State: Initialized, Role: FOLLOWER
I20260619 13:07:26.287726 5081 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } }
I20260619 13:07:26.287952 5081 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:26.288106 5081 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:26.288278 5081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:26.290491 5081 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } }
I20260619 13:07:26.291010 5081 leader_election.cc:302] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b9ae7f9f9e4432a86a6c079738ab9e; no voters:
I20260619 13:07:26.291626 5081 leader_election.cc:288] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:26.291819 5086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:26.292568 5086 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [term 1 LEADER]: Becoming Leader. State: Replica: e9b9ae7f9f9e4432a86a6c079738ab9e, State: Running, Role: LEADER
I20260619 13:07:26.293108 5086 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } }
I20260619 13:07:26.294004 5081 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:26.295970 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b9ae7f9f9e4432a86a6c079738ab9e. Latest consensus state: current_term: 1 leader_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } } }
I20260619 13:07:26.296033 5087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9ae7f9f9e4432a86a6c079738ab9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33191 } } }
I20260619 13:07:26.296478 5087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:26.298000 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:26.298638 5094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:26.303932 5094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:26.312969 5094 catalog_manager.cc:1347] Generated new cluster ID: 0eb0b52914cc49c084a0181606647354
I20260619 13:07:26.313118 5094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:26.334950 5094 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:26.335398 5094 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:26.343489 5094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: Generated new TSK 0
I20260619 13:07:26.344326 5094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:07:26.355468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--builtin_ntp_servers=127.0.97.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:07:26.702383 5105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:26.702667 5105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:26.702867 5105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:26.710255 5105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:26.710352 5105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:26.710532 5105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:26.720009 5105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:26.721318 5105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:26.723176 5105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:26.736984 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:26.737087 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:26.737253 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:26.738299 5105 server_base.cc:1034] running on GCE node
I20260619 13:07:26.738956 5105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:26.739987 5105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:26.741199 5105 hybrid_clock.cc:648] HybridClock initialized: now 1781874446741160 us; error 32 us; skew 500 ppm
I20260619 13:07:26.741472 5105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:26.743644 5105 webserver.cc:460] Webserver started at http://127.0.97.1:42047/ using document root <none> and password file <none>
I20260619 13:07:26.744279 5105 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:26.744364 5105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:26.744619 5105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:26.747586 5105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "85e647e2d154437081c5322a306d68ed"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bc7148f5e46bf931a9b45ec9fad3a588"
server_key_iv: "01329ae358dcff02108d726aa49a8c48"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.748332 5105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "85e647e2d154437081c5322a306d68ed"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bc7148f5e46bf931a9b45ec9fad3a588"
server_key_iv: "01329ae358dcff02108d726aa49a8c48"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.753319 5105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 13:07:26.756771 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:26.758236 5105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:07:26.758378 5105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "85e647e2d154437081c5322a306d68ed"
format_stamp: "Formatted at 2026-06-19 13:07:26 on dist-test-slave-q11v"
server_key: "bc7148f5e46bf931a9b45ec9fad3a588"
server_key_iv: "01329ae358dcff02108d726aa49a8c48"
server_key_version: "encryptionkey@0"
I20260619 13:07:26.758566 5105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:26.764103 5105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:26.764847 5105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:26.765126 5105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:26.766155 5105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:26.768124 5105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:26.768245 5105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:26.768370 5105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:26.768453 5105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:26.803205 5105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45245
I20260619 13:07:26.803232 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45245 every 8 connection(s)
I20260619 13:07:26.804541 5105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:26.807426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5105
I20260619 13:07:26.807614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:26.808461 388 external_mini_cluster.cc:1428] Setting key 965b62dfce41d31b839e74e3d0f98fa2
I20260619 13:07:26.811211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--builtin_ntp_servers=127.0.97.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 13:07:26.813264 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:26.807059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46981 (local address 127.0.97.62:33191)
0619 13:07:26.807741 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:26.807745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:26.809025 (+ 1280us) server_negotiation.cc:407] Connection header received
0619 13:07:26.810930 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:26.810935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:26.810995 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:26.811115 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:26.812504 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:26.812510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:26.812515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:26.812534 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:26.812583 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:26.812594 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:26.812717 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:26.812911 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 13:07:26.812975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":496,"threads_started":1}
I20260619 13:07:26.814289 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:26.807847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33191 (local address 127.0.97.1:46981)
0619 13:07:26.808870 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:26.808912 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:26.810677 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:26.811395 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:26.811409 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:26.811517 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:26.812208 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:26.812211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:26.812248 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:26.812334 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:26.812686 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:26.812698 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:26.812707 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:07:26.812996 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 13:07:26.813258 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260619 13:07:26.815862 5232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33191
I20260619 13:07:26.816205 5232 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:26.816903 5232 heartbeater.cc:511] Master 127.0.97.62:33191 requested a full tablet report, sending...
I20260619 13:07:26.819242 5046 ts_manager.cc:194] Registered new tserver with Master: 85e647e2d154437081c5322a306d68ed (127.0.97.1:45245)
W20260619 13:07:27.134465 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:27.134728 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:27.134816 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:27.142045 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:27.142143 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:27.142246 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:27.151382 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:27.152617 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:27.154420 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:27.167680 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:27.167802 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:27.167783 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:27.168435 5236 server_base.cc:1034] running on GCE node
I20260619 13:07:27.169090 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:27.170163 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:27.171408 5236 hybrid_clock.cc:648] HybridClock initialized: now 1781874447171343 us; error 56 us; skew 500 ppm
I20260619 13:07:27.171754 5236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:27.174077 5236 webserver.cc:460] Webserver started at http://127.0.97.2:44589/ using document root <none> and password file <none>
I20260619 13:07:27.174784 5236 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:27.174916 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:27.175233 5236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:27.178145 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "2cace59c608441fcad4ac6cdc03c850e"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "8c7eebfa10f7b7fa7c6c05636de7501e"
server_key_iv: "244a75ac5e3c1edddbfc2b350864a6d6"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.178943 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2cace59c608441fcad4ac6cdc03c850e"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "8c7eebfa10f7b7fa7c6c05636de7501e"
server_key_iv: "244a75ac5e3c1edddbfc2b350864a6d6"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.184114 5236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:27.187399 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.188679 5236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:07:27.188849 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "2cace59c608441fcad4ac6cdc03c850e"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "8c7eebfa10f7b7fa7c6c05636de7501e"
server_key_iv: "244a75ac5e3c1edddbfc2b350864a6d6"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.189075 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:27.194851 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:27.195649 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:27.195911 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:27.196772 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:27.198143 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:27.198266 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.198385 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:27.198469 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.230825 5236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36427
I20260619 13:07:27.230877 5362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36427 every 8 connection(s)
I20260619 13:07:27.232184 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:27.240104 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.234838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52795 (local address 127.0.97.62:33191)
0619 13:07:27.235014 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.235019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.236451 (+ 1432us) server_negotiation.cc:407] Connection header received
0619 13:07:27.238121 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.238125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.238180 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.238269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.239506 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.239509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.239513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.239529 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.239559 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.239569 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.239663 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.239825 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.239874 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:27.241053 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.235380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33191 (local address 127.0.97.2:52795)
0619 13:07:27.236278 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.236321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.237878 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.238477 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.238485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.238575 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.239219 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.239221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.239258 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.239340 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.239653 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.239660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.239665 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.239871 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.240136 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":278,"threads_started":1}
I20260619 13:07:27.241699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5236
I20260619 13:07:27.241884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:27.242395 388 external_mini_cluster.cc:1428] Setting key a654c1d03add9dd056462f4947cd7a34
I20260619 13:07:27.242507 5363 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33191
I20260619 13:07:27.242798 5363 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:27.243332 5363 heartbeater.cc:511] Master 127.0.97.62:33191 requested a full tablet report, sending...
I20260619 13:07:27.244604 5046 ts_manager.cc:194] Registered new tserver with Master: 2cace59c608441fcad4ac6cdc03c850e (127.0.97.2:36427)
I20260619 13:07:27.244975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--builtin_ntp_servers=127.0.97.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:07:27.591815 5366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:27.592119 5366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:27.592296 5366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:27.599691 5366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:27.599792 5366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:27.600018 5366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:27.609424 5366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:27.610678 5366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:27.612335 5366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:27.632269 5372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:27.632453 5374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:27.632328 5371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:27.633200 5366 server_base.cc:1034] running on GCE node
I20260619 13:07:27.633934 5366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:27.634958 5366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:27.636199 5366 hybrid_clock.cc:648] HybridClock initialized: now 1781874447636147 us; error 48 us; skew 500 ppm
I20260619 13:07:27.636541 5366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:27.638999 5366 webserver.cc:460] Webserver started at http://127.0.97.3:39085/ using document root <none> and password file <none>
I20260619 13:07:27.639724 5366 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:27.639858 5366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:27.640178 5366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:27.643211 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "f6ef5dc2a8354f7f9f8059165ada4b04"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "a9bb2aa2f89341230065e5f941f22871"
server_key_iv: "b69566166f714e563c5adcb3906b3444"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.644022 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ef5dc2a8354f7f9f8059165ada4b04"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "a9bb2aa2f89341230065e5f941f22871"
server_key_iv: "b69566166f714e563c5adcb3906b3444"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.649204 5366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:07:27.652745 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.654302 5366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:27.654492 5366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "f6ef5dc2a8354f7f9f8059165ada4b04"
format_stamp: "Formatted at 2026-06-19 13:07:27 on dist-test-slave-q11v"
server_key: "a9bb2aa2f89341230065e5f941f22871"
server_key_iv: "b69566166f714e563c5adcb3906b3444"
server_key_version: "encryptionkey@0"
I20260619 13:07:27.654749 5366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:27.660552 5366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:27.661368 5366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:27.661753 5366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:27.662686 5366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:27.664196 5366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:27.664281 5366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.664366 5366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:27.664419 5366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:27.697361 5366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36673
I20260619 13:07:27.697508 5492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36673 every 8 connection(s)
I20260619 13:07:27.698835 5366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:27.704476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5366
I20260619 13:07:27.704643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:27.705188 388 external_mini_cluster.cc:1428] Setting key 83910088d2b96b092a4fcfd36bd8025b
I20260619 13:07:27.708056 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.701383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37743 (local address 127.0.97.62:33191)
0619 13:07:27.701565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.701570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.703157 (+ 1587us) server_negotiation.cc:407] Connection header received
0619 13:07:27.705341 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.705345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.705402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.705509 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.707311 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.707315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.707318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.707337 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.707368 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.707379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.707526 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.707703 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.707763 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:07:27.709050 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.702057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33191 (local address 127.0.97.3:37743)
0619 13:07:27.703005 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.703040 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.704977 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.705846 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.705856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.705983 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.706873 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.706877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.706938 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.707057 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.707510 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.707518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.707525 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.707745 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.707956 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260619 13:07:27.710399 5493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33191
I20260619 13:07:27.710723 5493 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:27.711285 5493 heartbeater.cc:511] Master 127.0.97.62:33191 requested a full tablet report, sending...
I20260619 13:07:27.712596 5046 ts_manager.cc:194] Registered new tserver with Master: f6ef5dc2a8354f7f9f8059165ada4b04 (127.0.97.3:36673)
I20260619 13:07:27.720687 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:07:27.727320 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.724970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.97.62:33191)
0619 13:07:27.725191 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.725200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.725586 (+ 386us) server_negotiation.cc:407] Connection header received
0619 13:07:27.725782 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.725786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.725836 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.725914 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.726722 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.726726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.726729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.726744 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.726773 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.726784 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.726879 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.726974 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.727018 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:07:27.731060 5046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:27.737977 5046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:27.755983 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.750287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36673 (local address 127.0.0.1:50980)
0619 13:07:27.751164 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.751211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.751337 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.752477 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.752484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.752581 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.753489 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.753496 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.753541 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.753674 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.755117 (+ 1443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.755128 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.755137 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.755311 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.755422 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":412,"threads_started":1}
I20260619 13:07:27.756109 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.751703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36427 (local address 127.0.0.1:56452)
0619 13:07:27.752209 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.752225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.752310 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.753919 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.753927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.753958 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.754755 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.754761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.754782 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.754838 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.755384 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.755390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.755395 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.755544 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.755602 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":236,"threads_started":1}
I20260619 13:07:27.756886 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.752038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.97.2:36427)
0619 13:07:27.753354 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.753363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.753394 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:07:27.753463 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.753470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.753730 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.753925 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.755059 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.755067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.755079 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.755110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.755185 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.755211 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.756367 (+ 1156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.756447 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.756540 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":1006,"threads_started":1}
I20260619 13:07:27.757182 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.750581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.97.3:36673)
0619 13:07:27.751920 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.751931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.751970 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:27.752045 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.752064 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.752319 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.752483 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.754858 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.754871 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.754890 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.754909 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.754966 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.754987 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.755942 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.756026 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.756525 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":957,"threads_started":1}
I20260619 13:07:27.758611 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.752259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49174)
0619 13:07:27.755610 (+ 3351us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.755630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.755738 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.756561 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.756566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.756589 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.757405 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.757408 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.757425 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.757486 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.757991 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.757995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.758000 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.758131 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.758185 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"thread_start_us":3077,"threads_started":1}
I20260619 13:07:27.759321 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.752837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.0.97.1:45245)
0619 13:07:27.755187 (+ 2350us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.755196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.755744 (+ 548us) server_negotiation.cc:407] Connection header received
0619 13:07:27.755907 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.755916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.756154 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.756311 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.757688 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.757699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.757715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.757735 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.757794 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.757820 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.757941 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.758887 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.758997 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":2067,"threads_started":1}
I20260619 13:07:27.760980 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 (DEFAULT_TABLE table=test-table [id=e4341be264c9498abcfec3aa6b54ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:27.760980 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 (DEFAULT_TABLE table=test-table [id=e4341be264c9498abcfec3aa6b54ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:27.762591 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:27.762616 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:27.764423 5167 tablet_service.cc:1467] Processing CreateTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 (DEFAULT_TABLE table=test-table [id=e4341be264c9498abcfec3aa6b54ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:27.765574 5167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:27.778605 5513 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Bootstrap starting.
I20260619 13:07:27.781253 5511 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Bootstrap starting.
I20260619 13:07:27.782474 5512 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Bootstrap starting.
I20260619 13:07:27.784271 5513 tablet_bootstrap.cc:654] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:27.785496 5512 tablet_bootstrap.cc:654] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:27.785616 5511 tablet_bootstrap.cc:654] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:27.785667 5513 log.cc:824] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:27.786648 5512 log.cc:824] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:27.786805 5511 log.cc:824] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:27.789119 5513 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: No bootstrap required, opened a new log
I20260619 13:07:27.789422 5513 ts_tablet_manager.cc:1397] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 13:07:27.789464 5511 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: No bootstrap required, opened a new log
I20260619 13:07:27.789705 5512 tablet_bootstrap.cc:492] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: No bootstrap required, opened a new log
I20260619 13:07:27.789804 5511 ts_tablet_manager.cc:1397] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:07:27.789953 5512 ts_tablet_manager.cc:1397] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 13:07:27.797658 5511 raft_consensus.cc:348] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.798086 5511 raft_consensus.cc:374] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:27.798187 5511 raft_consensus.cc:729] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e647e2d154437081c5322a306d68ed, State: Initialized, Role: FOLLOWER
I20260619 13:07:27.798834 5511 consensus_queue.cc:260] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.799973 5232 heartbeater.cc:503] Master 127.0.97.62:33191 was elected leader, sending a full tablet report...
I20260619 13:07:27.800673 5511 ts_tablet_manager.cc:1428] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 13:07:27.800549 5512 raft_consensus.cc:348] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.800549 5513 raft_consensus.cc:348] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.800982 5512 raft_consensus.cc:374] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:27.800969 5513 raft_consensus.cc:374] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:27.801069 5512 raft_consensus.cc:729] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cace59c608441fcad4ac6cdc03c850e, State: Initialized, Role: FOLLOWER
I20260619 13:07:27.801070 5513 raft_consensus.cc:729] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ef5dc2a8354f7f9f8059165ada4b04, State: Initialized, Role: FOLLOWER
I20260619 13:07:27.801764 5513 consensus_queue.cc:260] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.801764 5512 consensus_queue.cc:260] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.803619 5512 ts_tablet_manager.cc:1428] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 13:07:27.804078 5363 heartbeater.cc:503] Master 127.0.97.62:33191 was elected leader, sending a full tablet report...
W20260619 13:07:27.806953 5233 tablet.cc:2369] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:27.807938 5493 heartbeater.cc:503] Master 127.0.97.62:33191 was elected leader, sending a full tablet report...
I20260619 13:07:27.808586 5513 ts_tablet_manager.cc:1428] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260619 13:07:27.822999 5519 raft_consensus.cc:482] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:27.823299 5519 raft_consensus.cc:504] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.824896 5519 leader_election.cc:288] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cace59c608441fcad4ac6cdc03c850e (127.0.97.2:36427), 85e647e2d154437081c5322a306d68ed (127.0.97.1:45245)
I20260619 13:07:27.827589 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.825345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36427 (local address 127.0.97.3:56417)
0619 13:07:27.825546 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.825566 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.825703 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.826060 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.826065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.826090 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.826776 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.826779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.826793 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.826832 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.827153 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.827158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.827162 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.827261 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.827316 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:07:27.827589 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.825375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56417 (local address 127.0.97.2:36427)
0619 13:07:27.825536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.825541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.825663 (+ 122us) server_negotiation.cc:407] Connection header received
0619 13:07:27.825829 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.825835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.825894 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.826036 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.826985 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.826990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.826995 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.827013 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.827049 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.827061 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.827179 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.827258 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.827316 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:07:27.828751 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.826044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57007 (local address 127.0.97.1:45245)
0619 13:07:27.826273 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.826281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.826867 (+ 586us) server_negotiation.cc:407] Connection header received
0619 13:07:27.827032 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.827036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.827081 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.827181 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.828245 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.828249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.828252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.828266 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.828295 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.828306 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.828396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.828524 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.828567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 13:07:27.828719 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a3dc4e98b4f0a8ddccd6da5ec0839" candidate_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cace59c608441fcad4ac6cdc03c850e" is_pre_election: true
I20260619 13:07:27.829149 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.825782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.97.3:57007)
0619 13:07:27.826762 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:27.826776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:27.826896 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:27.827199 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:27.827203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:27.827226 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:27.828063 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:07:27.828065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:27.828078 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:07:27.828110 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:27.828406 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:27.828410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:27.828415 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:07:27.828546 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:07:27.828599 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":691,"threads_started":1}
I20260619 13:07:27.829321 5318 raft_consensus.cc:2454] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ef5dc2a8354f7f9f8059165ada4b04 in term 0.
I20260619 13:07:27.830066 5382 leader_election.cc:302] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cace59c608441fcad4ac6cdc03c850e, f6ef5dc2a8354f7f9f8059165ada4b04; no voters:
I20260619 13:07:27.830163 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a3dc4e98b4f0a8ddccd6da5ec0839" candidate_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e647e2d154437081c5322a306d68ed" is_pre_election: true
I20260619 13:07:27.830466 5519 raft_consensus.cc:2781] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:27.830615 5519 raft_consensus.cc:482] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:27.830721 5519 raft_consensus.cc:3037] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:27.830775 5187 raft_consensus.cc:2454] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ef5dc2a8354f7f9f8059165ada4b04 in term 0.
I20260619 13:07:27.833197 5519 raft_consensus.cc:504] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.833964 5519 leader_election.cc:288] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [CANDIDATE]: Term 1 election: Requested vote from peers 2cace59c608441fcad4ac6cdc03c850e (127.0.97.2:36427), 85e647e2d154437081c5322a306d68ed (127.0.97.1:45245)
I20260619 13:07:27.834470 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a3dc4e98b4f0a8ddccd6da5ec0839" candidate_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cace59c608441fcad4ac6cdc03c850e"
I20260619 13:07:27.834690 5318 raft_consensus.cc:3037] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:27.834795 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713a3dc4e98b4f0a8ddccd6da5ec0839" candidate_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e647e2d154437081c5322a306d68ed"
I20260619 13:07:27.835031 5187 raft_consensus.cc:3037] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:27.837154 5187 raft_consensus.cc:2454] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ef5dc2a8354f7f9f8059165ada4b04 in term 1.
I20260619 13:07:27.837159 5318 raft_consensus.cc:2454] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ef5dc2a8354f7f9f8059165ada4b04 in term 1.
I20260619 13:07:27.837644 5382 leader_election.cc:302] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cace59c608441fcad4ac6cdc03c850e, f6ef5dc2a8354f7f9f8059165ada4b04; no voters:
I20260619 13:07:27.837960 5519 raft_consensus.cc:2781] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:27.838390 5519 raft_consensus.cc:686] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 1 LEADER]: Becoming Leader. State: Replica: f6ef5dc2a8354f7f9f8059165ada4b04, State: Running, Role: LEADER
I20260619 13:07:27.838855 5519 consensus_queue.cc:237] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260619 13:07:27.843564 5046 catalog_manager.cc:5495] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ef5dc2a8354f7f9f8059165ada4b04 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ef5dc2a8354f7f9f8059165ada4b04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:27.865717 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.863723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.97.3:36673)
0619 13:07:27.863893 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.863898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.864027 (+ 129us) server_negotiation.cc:407] Connection header received
0619 13:07:27.864196 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.864199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.864277 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.864361 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.865219 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.865222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.865225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.865236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.865259 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.865269 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.865366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.865433 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.865476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:07:27.873898 5318 raft_consensus.cc:1261] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 1 FOLLOWER]: Refusing update from remote peer f6ef5dc2a8354f7f9f8059165ada4b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:27.873898 5187 raft_consensus.cc:1261] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed [term 1 FOLLOWER]: Refusing update from remote peer f6ef5dc2a8354f7f9f8059165ada4b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:27.874903 5521 consensus_queue.cc:1035] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e647e2d154437081c5322a306d68ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:27.875304 5519 consensus_queue.cc:1035] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cace59c608441fcad4ac6cdc03c850e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:27.886799 5524 mvcc.cc:205] Tried to move back new op lower bound from 7298557738481647616 to 7298557738351931392. Current Snapshot: MvccSnapshot[applied={T|T < 7298557738481647616}]
I20260619 13:07:27.887964 5525 mvcc.cc:205] Tried to move back new op lower bound from 7298557738481647616 to 7298557738351931392. Current Snapshot: MvccSnapshot[applied={T|T < 7298557738481647616}]
I20260619 13:07:27.889209 5526 mvcc.cc:205] Tried to move back new op lower bound from 7298557738481647616 to 7298557738351931392. Current Snapshot: MvccSnapshot[applied={T|T < 7298557738481647616}]
I20260619 13:07:27.906082 5046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:27.906345 5046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:27.909843 5046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b9ae7f9f9e4432a86a6c079738ab9e: Sending DeleteTablet for 3 replicas of tablet 713a3dc4e98b4f0a8ddccd6da5ec0839
I20260619 13:07:27.910992 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:27 UTC) from {username='slave'} at 127.0.0.1:50980
I20260619 13:07:27.911170 5167 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:27 UTC) from {username='slave'} at 127.0.0.1:49174
I20260619 13:07:27.911697 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:27 UTC) from {username='slave'} at 127.0.0.1:56452
I20260619 13:07:27.913343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5105
I20260619 13:07:27.923610 5536 tablet_replica.cc:330] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: stopping tablet replica
I20260619 13:07:27.924135 5536 raft_consensus.cc:2229] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:27.924401 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:27.914436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.97.1:45245)
0619 13:07:27.914616 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:27.914621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:27.920297 (+ 5676us) server_negotiation.cc:407] Connection header received
0619 13:07:27.921301 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:27.921310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:27.921442 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:27.921545 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:27.923367 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:27.923373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:27.923377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:27.923399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:27.923435 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:27.923463 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:27.923560 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:27.924028 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 13:07:27.924090 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:07:27.924669 5536 raft_consensus.cc:2258] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:27.925552 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:27.927287 5536 ts_tablet_manager.cc:1905] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:27.938216 5536 ts_tablet_manager.cc:1918] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:27.939637 5536 log.cc:1196] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/713a3dc4e98b4f0a8ddccd6da5ec0839
I20260619 13:07:27.940423 5536 ts_tablet_manager.cc:1939] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 2cace59c608441fcad4ac6cdc03c850e: Deleting consensus metadata
I20260619 13:07:27.942464 5031 catalog_manager.cc:4841] TS 2cace59c608441fcad4ac6cdc03c850e (127.0.97.2:36427): tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 (table test-table [id=e4341be264c9498abcfec3aa6b54ff84]) successfully deleted
I20260619 13:07:27.944103 5534 tablet_replica.cc:330] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: stopping tablet replica
I20260619 13:07:27.944569 5534 raft_consensus.cc:2229] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:27.945322 5534 raft_consensus.cc:2258] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:27.947422 5534 ts_tablet_manager.cc:1905] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:27.956236 5534 ts_tablet_manager.cc:1918] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:27.956461 5534 log.cc:1196] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/713a3dc4e98b4f0a8ddccd6da5ec0839
I20260619 13:07:27.957010 5534 ts_tablet_manager.cc:1939] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P f6ef5dc2a8354f7f9f8059165ada4b04: Deleting consensus metadata
I20260619 13:07:27.958874 5031 catalog_manager.cc:4841] TS f6ef5dc2a8354f7f9f8059165ada4b04 (127.0.97.3:36673): tablet 713a3dc4e98b4f0a8ddccd6da5ec0839 (table test-table [id=e4341be264c9498abcfec3aa6b54ff84]) successfully deleted
I20260619 13:07:28.031410 5535 tablet_replica.cc:330] T 713a3dc4e98b4f0a8ddccd6da5ec0839 P 85e647e2d154437081c5322a306d68ed: stopping tablet replica
I20260619 13:07:28.031754 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5105
I20260619 13:07:28.048713 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5236
W20260619 13:07:28.048707 5031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45245 (error 108)
I20260619 13:07:28.050850 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.050349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49192)
0619 13:07:28.050541 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.050667 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 13:07:28.051299 5031 catalog_manager.cc:4584] TS 85e647e2d154437081c5322a306d68ed (127.0.97.1:45245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 713a3dc4e98b4f0a8ddccd6da5ec0839: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
I20260619 13:07:28.052296 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.050002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.0.97.2:36427)
0619 13:07:28.050180 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:28.050186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:28.050205 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:07:28.050270 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:28.050275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:28.050378 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:28.050493 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:28.051624 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:28.051631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:28.051636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:28.051655 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:28.051696 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:28.051712 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:28.051834 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:28.051957 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:07:28.052016 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:07:28.052985 5358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:28.105003 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.104445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49198)
0619 13:07:28.104689 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.104800 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 13:07:28.106155 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.105771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49214)
0619 13:07:28.105923 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.106020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:07:28.167151 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.166496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49222)
0619 13:07:28.166771 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.166911 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 13:07:28.167810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5236
I20260619 13:07:28.170012 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.169543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49232)
0619 13:07:28.169734 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.169831 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:07:28.184850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5366
I20260619 13:07:28.187855 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.185686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.0.97.3:36673)
0619 13:07:28.185854 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:28.185862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:28.185886 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:07:28.186130 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:28.186139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:28.186197 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:28.186303 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:28.187201 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:28.187211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:28.187216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:28.187233 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:28.187271 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:28.187283 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:28.187425 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:28.187511 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 13:07:28.187563 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:28.188519 5488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:28.251089 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.250566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49236)
0619 13:07:28.250799 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.250909 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 13:07:28.252267 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.251838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:49238)
0619 13:07:28.252073 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.252144 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 13:07:28.316841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5366
I20260619 13:07:28.336470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5015
I20260619 13:07:28.337767 5076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:28.441939 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.441263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:51398)
0619 13:07:28.441453 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:28.441697 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:07:28.442117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5015
2026-06-19T13:07:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2659 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 13:07:28.483433 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:07:28.486114 388 webserver.cc:460] Webserver started at https://127.0.0.1:42157/ using document root <none> and password file <none>
I20260619 13:07:28.486425 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:07:28.487076 388 webserver.cc:460] Webserver started at http://127.0.0.1:40715/ using document root <none> and password file <none>
2026-06-19T13:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:28Z Disabled control of system clock
I20260619 13:07:28.513279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37205
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37205
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/testchainca.cert with env {}
W20260619 13:07:28.833261 5548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:28.833559 5548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:28.833722 5548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:28.840863 5548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:28.840968 5548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:28.841005 5548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:28.841069 5548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:28.841114 5548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:07:28.841173 5548 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:07:28.841213 5548 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42157/jwks.json
I20260619 13:07:28.851229 5548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:28.852488 5548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:28.853780 5548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:28.865589 5554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:28.865644 5553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:28.866248 5548 server_base.cc:1034] running on GCE node
W20260619 13:07:28.866381 5556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:28.867223 5548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:28.869051 5548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:28.870261 5548 hybrid_clock.cc:648] HybridClock initialized: now 1781874448870221 us; error 61 us; skew 500 ppm
I20260619 13:07:28.870600 5548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:28.872751 5548 webserver.cc:460] Webserver started at http://127.0.97.62:39921/ using document root <none> and password file <none>
I20260619 13:07:28.873456 5548 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:28.873626 5548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:28.873937 5548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:28.876734 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "d3c6a866aadd4246aebe79d6d465357e"
format_stamp: "Formatted at 2026-06-19 13:07:28 on dist-test-slave-q11v"
server_key: "ca69b359ffcebeb0d0ce07b05040e883"
server_key_iv: "4b743a6486b4d0eecb9e4a94dffa56d3"
server_key_version: "encryptionkey@0"
I20260619 13:07:28.877547 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "d3c6a866aadd4246aebe79d6d465357e"
format_stamp: "Formatted at 2026-06-19 13:07:28 on dist-test-slave-q11v"
server_key: "ca69b359ffcebeb0d0ce07b05040e883"
server_key_iv: "4b743a6486b4d0eecb9e4a94dffa56d3"
server_key_version: "encryptionkey@0"
I20260619 13:07:28.882975 5548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:07:28.886476 5562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:28.887845 5548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:28.888029 5548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "d3c6a866aadd4246aebe79d6d465357e"
format_stamp: "Formatted at 2026-06-19 13:07:28 on dist-test-slave-q11v"
server_key: "ca69b359ffcebeb0d0ce07b05040e883"
server_key_iv: "4b743a6486b4d0eecb9e4a94dffa56d3"
server_key_version: "encryptionkey@0"
I20260619 13:07:28.888247 5548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:28.902678 5548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:28.903447 5548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:28.903730 5548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:28.921303 5548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37205
I20260619 13:07:28.921324 5613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37205 every 8 connection(s)
I20260619 13:07:28.922672 5548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:28.926244 5614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:28.931455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5548
I20260619 13:07:28.931651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:28.932343 388 external_mini_cluster.cc:1428] Setting key e0439973d5e4949afae42d9a7a6ac2a9
I20260619 13:07:28.937146 5614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e: Bootstrap starting.
I20260619 13:07:28.940654 5614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:28.941886 5614 log.cc:824] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:28.945150 5614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e: No bootstrap required, opened a new log
I20260619 13:07:28.946671 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:28.934215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.97.62:37205)
0619 13:07:28.935184 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:28.935194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:28.935234 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:07:28.936795 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:28.936825 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:28.937284 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:28.937666 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:28.938234 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:28.939512 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:28.941381 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:28.942131 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:28.942268 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:28.943720 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:28.943732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:28.943751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:28.944479 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:28.944597 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:28.944622 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:28.944748 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:28.944966 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 13:07:28.945306 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260619 13:07:28.950608 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:07:28.954138 5614 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } }
I20260619 13:07:28.954484 5614 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:28.954603 5614 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c6a866aadd4246aebe79d6d465357e, State: Initialized, Role: FOLLOWER
I20260619 13:07:28.955137 5614 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } }
I20260619 13:07:28.955353 5614 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:28.955476 5614 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:28.955615 5614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:28.958096 5614 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } }
I20260619 13:07:28.958614 5614 leader_election.cc:302] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c6a866aadd4246aebe79d6d465357e; no voters:
I20260619 13:07:28.959172 5614 leader_election.cc:288] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:28.959268 5619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:28.959995 5619 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [term 1 LEADER]: Becoming Leader. State: Replica: d3c6a866aadd4246aebe79d6d465357e, State: Running, Role: LEADER
I20260619 13:07:28.960495 5619 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } }
I20260619 13:07:28.961388 5614 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:28.963629 5621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c6a866aadd4246aebe79d6d465357e. Latest consensus state: current_term: 1 leader_uuid: "d3c6a866aadd4246aebe79d6d465357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } } }
I20260619 13:07:28.963733 5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c6a866aadd4246aebe79d6d465357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c6a866aadd4246aebe79d6d465357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37205 } } }
I20260619 13:07:28.964030 5621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:28.964145 5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:28.966168 5624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:28.970949 5624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:28.980084 5624 catalog_manager.cc:1347] Generated new cluster ID: f28af4bd7017456390c90691865043c0
I20260619 13:07:28.980202 5624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:28.992995 5624 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:28.994840 5624 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:29.007077 5624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c6a866aadd4246aebe79d6d465357e: Generated new TSK 0
I20260619 13:07:29.008113 5624 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:07:29.046494 5644 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 13:07:29.046662 5644 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:07:29.047809 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:29.026113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.97.62:37205)
0619 13:07:29.026320 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:29.026326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:29.026824 (+ 498us) server_negotiation.cc:407] Connection header received
0619 13:07:29.027099 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:29.027106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:29.027208 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:29.027409 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:07:29.027922 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.029093 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.030591 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.031287 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.031458 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:29.031974 (+ 516us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:07:29.047135 (+ 15161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:07:29.047520 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 13:07:29.047932 5617 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 13:07:29.049100 5643 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:29.026024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37205 (local address 127.0.0.1:36076)
0619 13:07:29.026709 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:29.026730 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:29.026888 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:29.027442 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:29.027445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:29.027454 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:07:29.027757 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:29.027765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.029246 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.029249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:29.030451 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:29.030462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.031453 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.031513 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:29.031633 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:29.031696 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:07:29.047408 (+ 15712us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:07:29.047673 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37205: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":361,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:37205: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 13:07:29.050694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5548
I20260619 13:07:29.051801 5609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:29.142673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5548
2026-06-19T13:07:29Z chronyd exiting
I20260619 13:07:29.192456 388 test_util.cc:175] -----------------------------------------------
I20260619 13:07:29.192641 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874405339864-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 13:07:29.196494 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:07:29.198406 388 webserver.cc:460] Webserver started at https://127.0.0.1:42365/ using document root <none> and password file <none>
I20260619 13:07:29.198628 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:07:29.199260 388 webserver.cc:460] Webserver started at http://127.0.0.1:44715/ using document root <none> and password file <none>
2026-06-19T13:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:29Z Disabled control of system clock
I20260619 13:07:29.225449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42979
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42979
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/testchainca.cert with env {}
W20260619 13:07:29.542548 5651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:29.542842 5651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:29.543008 5651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:29.550750 5651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:29.550889 5651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:29.550958 5651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:29.551065 5651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:29.551151 5651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:07:29.551236 5651 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:07:29.551301 5651 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42365/jwks.json
I20260619 13:07:29.561667 5651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:29.563019 5651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:29.564358 5651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:29.575815 5657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:29.575906 5656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:29.575964 5659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:29.576710 5651 server_base.cc:1034] running on GCE node
I20260619 13:07:29.577410 5651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:29.578780 5651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:29.579972 5651 hybrid_clock.cc:648] HybridClock initialized: now 1781874449579941 us; error 28 us; skew 500 ppm
I20260619 13:07:29.580263 5651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:29.586249 5651 webserver.cc:460] Webserver started at http://127.0.97.62:34419/ using document root <none> and password file <none>
I20260619 13:07:29.586894 5651 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:29.587044 5651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:29.587339 5651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:29.590148 5651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "9ac6a29da8a74ec4913fa3481e4be82f"
format_stamp: "Formatted at 2026-06-19 13:07:29 on dist-test-slave-q11v"
server_key: "39af33d1892b8df65ab773bb7c0e6232"
server_key_iv: "7a132c80ae4e933f919c20657ce56b21"
server_key_version: "encryptionkey@0"
I20260619 13:07:29.590889 5651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ac6a29da8a74ec4913fa3481e4be82f"
format_stamp: "Formatted at 2026-06-19 13:07:29 on dist-test-slave-q11v"
server_key: "39af33d1892b8df65ab773bb7c0e6232"
server_key_iv: "7a132c80ae4e933f919c20657ce56b21"
server_key_version: "encryptionkey@0"
I20260619 13:07:29.595613 5651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:07:29.598881 5665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:29.600394 5651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:29.600615 5651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "9ac6a29da8a74ec4913fa3481e4be82f"
format_stamp: "Formatted at 2026-06-19 13:07:29 on dist-test-slave-q11v"
server_key: "39af33d1892b8df65ab773bb7c0e6232"
server_key_iv: "7a132c80ae4e933f919c20657ce56b21"
server_key_version: "encryptionkey@0"
I20260619 13:07:29.600850 5651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:29.618443 5651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:29.619315 5651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:29.619612 5651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:29.637708 5651 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42979
I20260619 13:07:29.637710 5716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42979 every 8 connection(s)
I20260619 13:07:29.639134 5651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:29.643059 5717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:29.644039 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5651
I20260619 13:07:29.644227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:29.645059 388 external_mini_cluster.cc:1428] Setting key 138519fba301a7dc709d599156244818
I20260619 13:07:29.656272 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f: Bootstrap starting.
I20260619 13:07:29.658591 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:29.647134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.0.97.62:42979)
0619 13:07:29.648200 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:29.648213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:29.648257 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:07:29.649961 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:29.649986 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:29.650509 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:29.650947 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:29.651502 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.652749 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.654101 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.654799 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.654992 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:29.656191 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:29.656198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:29.656213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:29.656814 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:29.656896 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:29.656916 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:29.657049 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:29.657271 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:07:29.657633 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":360,"threads_started":1}
I20260619 13:07:29.659646 5717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:29.660693 5717 log.cc:824] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:29.662452 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:07:29.664057 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f: No bootstrap required, opened a new log
I20260619 13:07:29.671522 5717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } }
I20260619 13:07:29.671975 5717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:29.672094 5717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac6a29da8a74ec4913fa3481e4be82f, State: Initialized, Role: FOLLOWER
I20260619 13:07:29.672724 5717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } }
I20260619 13:07:29.672940 5717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:29.673036 5717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:29.673159 5717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:29.675329 5717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } }
I20260619 13:07:29.675810 5717 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac6a29da8a74ec4913fa3481e4be82f; no voters:
I20260619 13:07:29.676311 5717 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:29.676486 5722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:29.677191 5722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [term 1 LEADER]: Becoming Leader. State: Replica: 9ac6a29da8a74ec4913fa3481e4be82f, State: Running, Role: LEADER
I20260619 13:07:29.677811 5722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } }
I20260619 13:07:29.678580 5717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:29.680325 5724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac6a29da8a74ec4913fa3481e4be82f. Latest consensus state: current_term: 1 leader_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } } }
I20260619 13:07:29.680727 5724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:29.680922 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac6a29da8a74ec4913fa3481e4be82f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42979 } } }
I20260619 13:07:29.681284 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:29.685461 5728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:29.690258 5728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:29.698793 5728 catalog_manager.cc:1347] Generated new cluster ID: fb095aa494f4441ea2ee687cce6cf780
I20260619 13:07:29.698925 5728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:29.718950 5728 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:29.720252 5728 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:29.728580 5728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac6a29da8a74ec4913fa3481e4be82f: Generated new TSK 0
I20260619 13:07:29.729378 5728 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:07:29.767980 5747 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 13:07:29.768119 5747 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:07:29.769349 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:29.742286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.97.62:42979)
0619 13:07:29.742491 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:29.742500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:29.742903 (+ 403us) server_negotiation.cc:407] Connection header received
0619 13:07:29.743183 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:29.743186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:29.743262 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:29.743426 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:07:29.743883 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.744602 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.746019 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.746467 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.746611 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:29.747089 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:07:29.768508 (+ 21419us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:07:29.768992 (+ 484us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 13:07:29.769490 5720 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 13:07:29.769515 5746 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:29.742169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42979 (local address 127.0.0.1:33444)
0619 13:07:29.742798 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:29.742814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:29.742980 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:29.743439 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:29.743444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:29.743456 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:07:29.743713 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:29.743722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.744760 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.744765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:29.745881 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:29.745891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:29.746598 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:29.746644 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:29.746802 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:29.746835 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:07:29.768810 (+ 21975us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:07:29.769076 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":352,"threads_started":1}
I20260619 13:07:29.770594 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5651
I20260619 13:07:29.772111 5712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:29.864418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5651
2026-06-19T13:07:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 13:07:29.919425 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:07:29.921177 388 webserver.cc:460] Webserver started at https://127.0.0.1:34517/ using document root <none> and password file <none>
I20260619 13:07:29.921386 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:07:29.921994 388 webserver.cc:460] Webserver started at http://127.0.0.1:39773/ using document root <none> and password file <none>
2026-06-19T13:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:29Z Disabled control of system clock
I20260619 13:07:29.948336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43729
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34517/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:07:30.268334 5754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:30.268632 5754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:30.268816 5754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:30.276218 5754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:30.276335 5754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:30.276422 5754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:30.276525 5754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:30.276605 5754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:07:30.276705 5754 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:07:30.276765 5754 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34517/jwks.json
I20260619 13:07:30.286151 5754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34517/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:30.287525 5754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:30.289430 5754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:30.300925 5759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:30.301005 5760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:30.300966 5762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:30.302275 5754 server_base.cc:1034] running on GCE node
I20260619 13:07:30.303059 5754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:30.304463 5754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:30.305696 5754 hybrid_clock.cc:648] HybridClock initialized: now 1781874450305656 us; error 52 us; skew 500 ppm
I20260619 13:07:30.306063 5754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:30.308296 5754 webserver.cc:460] Webserver started at http://127.0.97.62:33317/ using document root <none> and password file <none>
I20260619 13:07:30.309106 5754 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:30.309243 5754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:30.309587 5754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:30.312541 5754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "252db6406a1e4317b22cb3b1570b91af"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "1fe302d0540f1c61d0bf9005286f0c9f"
server_key_iv: "1255445d9cbe53099b9dc6364d6c302e"
server_key_version: "encryptionkey@0"
I20260619 13:07:30.313431 5754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "252db6406a1e4317b22cb3b1570b91af"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "1fe302d0540f1c61d0bf9005286f0c9f"
server_key_iv: "1255445d9cbe53099b9dc6364d6c302e"
server_key_version: "encryptionkey@0"
I20260619 13:07:30.318576 5754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:07:30.321974 5768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:30.323334 5754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:07:30.323519 5754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "252db6406a1e4317b22cb3b1570b91af"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "1fe302d0540f1c61d0bf9005286f0c9f"
server_key_iv: "1255445d9cbe53099b9dc6364d6c302e"
server_key_version: "encryptionkey@0"
I20260619 13:07:30.323736 5754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:30.339870 5754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:30.340692 5754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:30.340986 5754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:30.358862 5754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43729
I20260619 13:07:30.358873 5819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43729 every 8 connection(s)
I20260619 13:07:30.360219 5754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:30.363857 5820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:30.367112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5754
I20260619 13:07:30.367338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:30.367946 388 external_mini_cluster.cc:1428] Setting key 35c928fa7e25364bfa95ba2f024526b5
I20260619 13:07:30.375560 5820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af: Bootstrap starting.
I20260619 13:07:30.378803 5820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:30.379956 5820 log.cc:824] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:30.380719 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:30.369686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.97.62:43729)
0619 13:07:30.370638 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:30.370649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:30.370689 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:07:30.372241 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:30.372262 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:30.372732 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:30.373130 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:30.373718 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.374939 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.376249 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.376943 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.377147 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:30.378342 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:30.378348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:30.378362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:30.378881 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:30.379025 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:30.379046 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:30.379176 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:30.379392 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 13:07:30.379688 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
I20260619 13:07:30.382853 5820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af: No bootstrap required, opened a new log
I20260619 13:07:30.384693 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:07:30.391712 5820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } }
I20260619 13:07:30.392067 5820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:30.392174 5820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252db6406a1e4317b22cb3b1570b91af, State: Initialized, Role: FOLLOWER
I20260619 13:07:30.392796 5820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } }
I20260619 13:07:30.393052 5820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:30.393182 5820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:30.393359 5820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:30.395572 5820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } }
I20260619 13:07:30.396072 5820 leader_election.cc:302] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252db6406a1e4317b22cb3b1570b91af; no voters:
I20260619 13:07:30.396649 5820 leader_election.cc:288] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:30.396819 5825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:30.398108 5825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [term 1 LEADER]: Becoming Leader. State: Replica: 252db6406a1e4317b22cb3b1570b91af, State: Running, Role: LEADER
I20260619 13:07:30.398608 5825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } }
I20260619 13:07:30.399014 5820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:30.401257 5826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252db6406a1e4317b22cb3b1570b91af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } } }
I20260619 13:07:30.401585 5826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:30.401798 5827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252db6406a1e4317b22cb3b1570b91af. Latest consensus state: current_term: 1 leader_uuid: "252db6406a1e4317b22cb3b1570b91af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252db6406a1e4317b22cb3b1570b91af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43729 } } }
I20260619 13:07:30.402123 5827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:30.404415 5832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:30.409545 5832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:30.418910 5832 catalog_manager.cc:1347] Generated new cluster ID: 0860e587661d4e2c9677d27257fc1603
I20260619 13:07:30.419039 5832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:30.430737 5832 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:30.432520 5832 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:30.440304 5832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252db6406a1e4317b22cb3b1570b91af: Generated new TSK 0
I20260619 13:07:30.441363 5832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:07:30.467803 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:30.459385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.0.97.62:43729)
0619 13:07:30.459605 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:30.459612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:30.460019 (+ 407us) server_negotiation.cc:407] Connection header received
0619 13:07:30.460224 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:30.460230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:30.460320 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:30.460486 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:07:30.460932 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.462097 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.463409 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.464075 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.464195 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:30.464664 (+ 469us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:07:30.467141 (+ 2477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34517/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:07:30.467476 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34517/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 13:07:30.467936 5823 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34517/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 13:07:30.469516 5849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:30.459247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43729 (local address 127.0.0.1:42700)
0619 13:07:30.459913 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:30.459930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:30.460054 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:30.460512 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:30.460515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:30.460524 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:07:30.460786 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:30.460794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.462234 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.462239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:30.463270 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:30.463281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:30.464200 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:30.464257 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:30.464379 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:30.464410 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:07:30.467364 (+ 2954us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34517/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:07:30.469093 (+ 1729us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34517/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1411,"thread_start_us":365,"threads_started":1}
I20260619 13:07:30.470979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5754
I20260619 13:07:30.472200 5815 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 13:07:30.479756 5850 webserver.cc:533] Webserver: sslize failed:
E20260619 13:07:30.479878 5850 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:07:30.560495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5754
2026-06-19T13:07:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (695 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T13:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:30Z Disabled control of system clock
I20260619 13:07:30.643086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43079
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43079
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/testchainca.cert with env {}
W20260619 13:07:30.960227 5855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:30.960588 5855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:30.960750 5855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:30.967944 5855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:30.968039 5855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:30.968119 5855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:30.968237 5855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:30.968298 5855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:07:30.968377 5855 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:07:30.968431 5855 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 13:07:30.978238 5855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:30.979560 5855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:30.980954 5855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:30.992347 5860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:30.992470 5863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:30.992545 5861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:30.993333 5855 server_base.cc:1034] running on GCE node
I20260619 13:07:30.994159 5855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:30.995580 5855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:30.996806 5855 hybrid_clock.cc:648] HybridClock initialized: now 1781874450996753 us; error 49 us; skew 500 ppm
I20260619 13:07:30.997156 5855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:07:30.999562 5855 webserver.cc:460] Webserver started at http://127.0.97.62:39093/ using document root <none> and password file <none>
I20260619 13:07:31.000262 5855 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:31.000399 5855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:31.000706 5855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:31.003639 5855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "7291704ef0d443fa8ff58d0c40bbda49"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "4ac2094b51f5867358dc38dd8f337d25"
server_key_iv: "f0f4139f11da59221a5f864cc7da7773"
server_key_version: "encryptionkey@0"
I20260619 13:07:31.004458 5855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "7291704ef0d443fa8ff58d0c40bbda49"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "4ac2094b51f5867358dc38dd8f337d25"
server_key_iv: "f0f4139f11da59221a5f864cc7da7773"
server_key_version: "encryptionkey@0"
I20260619 13:07:31.009539 5855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:07:31.013322 5869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:31.014791 5855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:31.015021 5855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "7291704ef0d443fa8ff58d0c40bbda49"
format_stamp: "Formatted at 2026-06-19 13:07:30 on dist-test-slave-q11v"
server_key: "4ac2094b51f5867358dc38dd8f337d25"
server_key_iv: "f0f4139f11da59221a5f864cc7da7773"
server_key_version: "encryptionkey@0"
I20260619 13:07:31.015317 5855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:31.070021 5855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:31.070976 5855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:31.071290 5855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:31.089344 5855 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43079
I20260619 13:07:31.089382 5920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43079 every 8 connection(s)
I20260619 13:07:31.090862 5855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:31.092860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5855
I20260619 13:07:31.093065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:31.093786 388 external_mini_cluster.cc:1428] Setting key 60e823617bdfac5972f612f7a519570f
I20260619 13:07:31.095120 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:31.109411 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:31.095701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.62:43079)
0619 13:07:31.096766 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:31.096781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:31.096833 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:07:31.098663 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:31.098691 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:31.099329 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:31.099783 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:31.100332 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.102541 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.103944 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.104984 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.105188 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:31.106422 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:31.106432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:31.106452 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:07:31.107237 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:31.107339 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:07:31.107369 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:31.107492 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:31.107758 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 13:07:31.108132 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260619 13:07:31.110186 5921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49: Bootstrap starting.
I20260619 13:07:31.113616 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:07:31.113950 5921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:31.115128 5921 log.cc:824] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:31.118259 5921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49: No bootstrap required, opened a new log
I20260619 13:07:31.125919 5921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } }
I20260619 13:07:31.126328 5921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:31.126435 5921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7291704ef0d443fa8ff58d0c40bbda49, State: Initialized, Role: FOLLOWER
I20260619 13:07:31.126993 5921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } }
I20260619 13:07:31.127230 5921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:31.127377 5921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:31.127549 5921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:31.130105 5921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } }
I20260619 13:07:31.130631 5921 leader_election.cc:302] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7291704ef0d443fa8ff58d0c40bbda49; no voters:
I20260619 13:07:31.131232 5921 leader_election.cc:288] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:31.131376 5926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:31.132031 5926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [term 1 LEADER]: Becoming Leader. State: Replica: 7291704ef0d443fa8ff58d0c40bbda49, State: Running, Role: LEADER
I20260619 13:07:31.132581 5926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } }
I20260619 13:07:31.133343 5921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:31.135634 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7291704ef0d443fa8ff58d0c40bbda49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } } }
I20260619 13:07:31.135620 5928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7291704ef0d443fa8ff58d0c40bbda49. Latest consensus state: current_term: 1 leader_uuid: "7291704ef0d443fa8ff58d0c40bbda49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7291704ef0d443fa8ff58d0c40bbda49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43079 } } }
I20260619 13:07:31.136050 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:31.136108 5928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:31.142208 5934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:31.148737 5934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:31.157632 5934 catalog_manager.cc:1347] Generated new cluster ID: 61121550a0864b5aa626e0535cd20076
I20260619 13:07:31.157752 5934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:31.196753 5934 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:31.198612 5934 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:31.216683 5934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7291704ef0d443fa8ff58d0c40bbda49: Generated new TSK 0
I20260619 13:07:31.217615 5934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:07:31.313262 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:31.233468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.97.62:43079)
0619 13:07:31.233690 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:31.233697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:31.234112 (+ 415us) server_negotiation.cc:407] Connection header received
0619 13:07:31.234360 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:31.234366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:31.234488 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:31.234656 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:07:31.235136 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.236272 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.237746 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.238460 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.238589 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:31.239044 (+ 455us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:07:31.312523 (+ 73479us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:07:31.312945 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34318: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 13:07:31.313409 5924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34318: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 13:07:31.313562 5950 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:31.233277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43079 (local address 127.0.0.1:34318)
0619 13:07:31.234017 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:31.234041 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:31.234174 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:31.234689 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:31.234693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:31.234737 (+ 44us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:07:31.234987 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:31.234995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.236428 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.236432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:31.237531 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:31.237552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:31.238577 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:31.238622 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:31.238744 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:31.238777 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:07:31.312833 (+ 74056us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:07:31.313153 (+ 320us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":461,"threads_started":1}
I20260619 13:07:31.315018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5855
I20260619 13:07:31.316272 5916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:31.418871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5855
2026-06-19T13:07:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (836 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (408 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:32 dist-test-slave-q11v krb5kdc[5965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:32 dist-test-slave-q11v krb5kdc[5965](info): set up 2 sockets
Jun 19 13:07:32 dist-test-slave-q11v krb5kdc[5965](info): commencing operation
krb5kdc: starting...
W20260619 13:07:33.306380 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:33 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:33.515223 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37735
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:33.837414 5978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:33.837769 5978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:33.837960 5978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:33.845016 5978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:33.845120 5978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:33.845209 5978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:33.845296 5978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:33.845389 5978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:33.855222 5978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:33.856561 5978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:33.857929 5978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:33.869949 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:33.869997 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:33.870018 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:33.870786 5978 server_base.cc:1034] running on GCE node
I20260619 13:07:33.871614 5978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:33.873068 5978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:33.874294 5978 hybrid_clock.cc:648] HybridClock initialized: now 1781874453874249 us; error 66 us; skew 500 ppm
I20260619 13:07:33.874663 5978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:33 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:33.877702 5978 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:33.879227 5978 webserver.cc:460] Webserver started at http://127.0.97.62:37267/ using document root <none> and password file <none>
I20260619 13:07:33.879917 5978 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:33.880060 5978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:33.880381 5978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:33.883388 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "7db63dc744394c1e8ffeb2bad296d734"
format_stamp: "Formatted at 2026-06-19 13:07:33 on dist-test-slave-q11v"
server_key: "68c21f8fbfa235a390d550cc834bd99a"
server_key_iv: "1c051d53ef3033be635d62e4f4d1f45f"
server_key_version: "encryptionkey@0"
I20260619 13:07:33.884193 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "7db63dc744394c1e8ffeb2bad296d734"
format_stamp: "Formatted at 2026-06-19 13:07:33 on dist-test-slave-q11v"
server_key: "68c21f8fbfa235a390d550cc834bd99a"
server_key_iv: "1c051d53ef3033be635d62e4f4d1f45f"
server_key_version: "encryptionkey@0"
I20260619 13:07:33.889448 5978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:07:33.892896 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:33.894450 5978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:33.894657 5978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "7db63dc744394c1e8ffeb2bad296d734"
format_stamp: "Formatted at 2026-06-19 13:07:33 on dist-test-slave-q11v"
server_key: "68c21f8fbfa235a390d550cc834bd99a"
server_key_iv: "1c051d53ef3033be635d62e4f4d1f45f"
server_key_version: "encryptionkey@0"
I20260619 13:07:33.894879 5978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:33.933753 5978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:33.938258 5978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:33.938580 5978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:33.958012 5978 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37735
I20260619 13:07:33.958047 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37735 every 8 connection(s)
I20260619 13:07:33.959492 5978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:33.963192 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:33.966012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 5978
I20260619 13:07:33.966214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:33.966971 388 external_mini_cluster.cc:1428] Setting key 42e835a595881f89baff7ae6a961f3b0
I20260619 13:07:33.976584 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734: Bootstrap starting.
I20260619 13:07:33.979687 6045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:33.980691 6045 log.cc:824] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:07:33 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:33.983773 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734: No bootstrap required, opened a new log
I20260619 13:07:33.991943 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:33.969901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.0.97.62:37735)
0619 13:07:33.970935 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:33.970952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:33.971471 (+ 519us) server_negotiation.cc:407] Connection header received
0619 13:07:33.973272 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:33.973293 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:33.973793 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:33.974206 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:33.975618 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:33.976997 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:33.978313 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:33.978980 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:33.979166 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:33.983163 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:33.983174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:33.983193 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:33.983238 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:33.987362 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:33.988080 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:33.988093 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:33.988101 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:33.988307 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:33.988654 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:33.988659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:33.988663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:33.989341 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:33.989667 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:33.990081 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 13:07:33.990530 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":693,"threads_started":1}
I20260619 13:07:33.994536 6045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } }
I20260619 13:07:33.994971 6045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:33.995070 6045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db63dc744394c1e8ffeb2bad296d734, State: Initialized, Role: FOLLOWER
I20260619 13:07:33.995635 6045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } }
I20260619 13:07:33.995857 6045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:33.996006 6045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:33.996173 6045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:33.998636 6045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } }
I20260619 13:07:33.999114 6045 leader_election.cc:302] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db63dc744394c1e8ffeb2bad296d734; no voters:
I20260619 13:07:33.999758 6045 leader_election.cc:288] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:33.999907 6050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:34.000588 6050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [term 1 LEADER]: Becoming Leader. State: Replica: 7db63dc744394c1e8ffeb2bad296d734, State: Running, Role: LEADER
I20260619 13:07:34.001102 6050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } }
I20260619 13:07:34.001930 6045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:34.004838 6052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db63dc744394c1e8ffeb2bad296d734. Latest consensus state: current_term: 1 leader_uuid: "7db63dc744394c1e8ffeb2bad296d734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } } }
I20260619 13:07:34.005282 6052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:34.004818 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db63dc744394c1e8ffeb2bad296d734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db63dc744394c1e8ffeb2bad296d734" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37735 } } }
I20260619 13:07:34.005999 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:34.010747 6057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:34.015167 6057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:34.024366 6057 catalog_manager.cc:1347] Generated new cluster ID: 682dbeb1eb7540fb87447a154ea3233c
I20260619 13:07:34.024544 6057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:34.041779 6057 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:34.043176 6057 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:34.054724 6057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db63dc744394c1e8ffeb2bad296d734: Generated new TSK 0
I20260619 13:07:34.055757 6057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:34.113212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:34.443717 6071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:34.444034 6071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:34.444128 6071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:34.451471 6071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:34.451589 6071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:34.451702 6071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:34.461668 6071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:34.462963 6071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:34.464193 6071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:34.477367 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:34.477424 6079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:34.477499 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:34.478215 6071 server_base.cc:1034] running on GCE node
I20260619 13:07:34.478768 6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:34.479667 6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:34.480870 6071 hybrid_clock.cc:648] HybridClock initialized: now 1781874454480827 us; error 29 us; skew 500 ppm
I20260619 13:07:34.481197 6071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:34 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:34.484364 6071 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:34.485919 6071 webserver.cc:460] Webserver started at http://127.0.97.1:41385/ using document root <none> and password file <none>
I20260619 13:07:34.486613 6071 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:34.486760 6071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:34.487066 6071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:34.490041 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "dfbd3678935f4cc5998d18fe03b14cfc"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "6edbd34d023f0d745b01070694e19da9"
server_key_iv: "33f9e97eab6147e73abea51e3e3bca30"
server_key_version: "encryptionkey@0"
I20260619 13:07:34.491024 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dfbd3678935f4cc5998d18fe03b14cfc"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "6edbd34d023f0d745b01070694e19da9"
server_key_iv: "33f9e97eab6147e73abea51e3e3bca30"
server_key_version: "encryptionkey@0"
I20260619 13:07:34.496383 6071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:07:34.500146 6086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:34.501513 6071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:34.501730 6071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "dfbd3678935f4cc5998d18fe03b14cfc"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "6edbd34d023f0d745b01070694e19da9"
server_key_iv: "33f9e97eab6147e73abea51e3e3bca30"
server_key_version: "encryptionkey@0"
I20260619 13:07:34.501950 6071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:34.526412 6071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:34.530491 6071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:34.530802 6071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:34.531795 6071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:34.533197 6071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:34.533298 6071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:34.533440 6071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:34.533510 6071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:34.566455 6071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44741
I20260619 13:07:34.566545 6198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44741 every 8 connection(s)
I20260619 13:07:34.567764 6071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:34.573685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6071
I20260619 13:07:34.573879 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:34.574512 388 external_mini_cluster.cc:1428] Setting key 44f1f9672815275e712b2d2cbecbb783
Jun 19 13:07:34 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:34.588312 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:34.570481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44051 (local address 127.0.97.62:37735)
0619 13:07:34.570975 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:34.570980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:34.571959 (+ 979us) server_negotiation.cc:407] Connection header received
0619 13:07:34.573946 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:34.573951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:34.574045 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:34.574163 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:34.575820 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:34.576731 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:34.578365 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:34.578785 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:34.578921 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:34.582534 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:34.582539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:34.582543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:34.582573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:34.585075 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:34.586058 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:34.586062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:34.586065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:34.586175 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:34.586638 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:34.586641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:34.586643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:34.586897 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:34.587001 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:34.587889 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 13:07:34.588060 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260619 13:07:34.589972 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:34.571041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37735 (local address 127.0.97.1:44051)
0619 13:07:34.571782 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:34.571822 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:34.573577 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:34.574414 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:34.574423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:34.575007 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:34.575657 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:34.575674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:34.576874 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:34.576887 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:34.578223 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:34.578235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:34.579011 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:34.579019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:34.579256 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:34.579952 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:34.579974 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:34.582319 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:34.585218 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:34.585230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:34.585250 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:34.585941 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:34.586281 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:34.586288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:34.586294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:34.586533 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:34.587030 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:34.587041 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:34.587486 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 13:07:34.587797 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 13:07:34.588404 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
I20260619 13:07:34.591653 6199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37735
I20260619 13:07:34.591960 6199 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:34.592664 6199 heartbeater.cc:511] Master 127.0.97.62:37735 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:07:34.594789 6010 ts_manager.cc:194] Registered new tserver with Master: dfbd3678935f4cc5998d18fe03b14cfc (127.0.97.1:44741)
I20260619 13:07:34.596477 6010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:34.620950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:34.938459 6205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:34.938796 6205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:34.938964 6205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:34.946192 6205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:34.946344 6205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:34.946538 6205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:34.956187 6205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:34.957585 6205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:34.958956 6205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:34.972141 6211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:34.972493 6213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:34.972645 6210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:34.974349 6205 server_base.cc:1034] running on GCE node
I20260619 13:07:34.975049 6205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:34.976101 6205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:34.977306 6205 hybrid_clock.cc:648] HybridClock initialized: now 1781874454977269 us; error 30 us; skew 500 ppm
I20260619 13:07:34.977581 6205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:34 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:34.986578 6205 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:34.988135 6205 webserver.cc:460] Webserver started at http://127.0.97.2:33039/ using document root <none> and password file <none>
I20260619 13:07:34.988811 6205 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:34.988914 6205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:34.989171 6205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:34.992187 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "12e7e86cbc56490289e8acb612858b18"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "e79e987fb8c44a4e938647dc4931c5c8"
server_key_iv: "632505f2d58c171c6f1fe2ec85187322"
server_key_version: "encryptionkey@0"
I20260619 13:07:34.992908 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "12e7e86cbc56490289e8acb612858b18"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "e79e987fb8c44a4e938647dc4931c5c8"
server_key_iv: "632505f2d58c171c6f1fe2ec85187322"
server_key_version: "encryptionkey@0"
I20260619 13:07:34.998096 6205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 13:07:35.001715 6220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.003098 6205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:35.003281 6205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "12e7e86cbc56490289e8acb612858b18"
format_stamp: "Formatted at 2026-06-19 13:07:34 on dist-test-slave-q11v"
server_key: "e79e987fb8c44a4e938647dc4931c5c8"
server_key_iv: "632505f2d58c171c6f1fe2ec85187322"
server_key_version: "encryptionkey@0"
I20260619 13:07:35.003506 6205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:35.025560 6205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:35.030053 6205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:35.030305 6205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:35.031219 6205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:35.032498 6205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:35.032569 6205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.032653 6205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:35.032697 6205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.065207 6205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41117
I20260619 13:07:35.065279 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41117 every 8 connection(s)
I20260619 13:07:35.066531 6205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:35.070212 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6205
I20260619 13:07:35.070430 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:35.070993 388 external_mini_cluster.cc:1428] Setting key cdb4b25592ee6064b9ac6df6631befe2
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:35.089917 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.069096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40121 (local address 127.0.97.62:37735)
0619 13:07:35.069277 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.069282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.070760 (+ 1478us) server_negotiation.cc:407] Connection header received
0619 13:07:35.072983 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.072988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.073067 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.073165 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.075315 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.075983 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.078546 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.078941 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.079032 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.082910 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.082919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.082927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.082972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.086519 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.087261 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.087270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.087274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.087426 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.087843 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.087851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.087857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.088230 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.088378 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.089155 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.089413 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:07:35.090616 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.069711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37735 (local address 127.0.97.2:40121)
0619 13:07:35.070566 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:35.070619 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:35.072708 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:35.073453 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:35.073465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:35.074279 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:35.075139 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:35.075167 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.076149 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.076156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:35.078391 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:35.078409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.079165 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.079175 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:35.079502 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.080256 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:35.080279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:35.082649 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:35.086660 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:35.086667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:35.086683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:35.087118 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:35.087531 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:35.087534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:35.087537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:35.087712 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:35.088350 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:35.088356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:35.088720 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 13:07:35.089004 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 13:07:35.089408 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":322,"threads_started":1}
I20260619 13:07:35.092150 6333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37735
I20260619 13:07:35.092525 6333 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:07:35.093434 6333 heartbeater.cc:511] Master 127.0.97.62:37735 requested a full tablet report, sending...
I20260619 13:07:35.095033 6010 ts_manager.cc:194] Registered new tserver with Master: 12e7e86cbc56490289e8acb612858b18 (127.0.97.2:41117)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:07:35.096215 6010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:35.119025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:35.448016 6338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:35.448347 6338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:35.448536 6338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:35.456276 6338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:35.456406 6338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:35.456705 6338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:35.466868 6338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:35.468219 6338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:35.469554 6338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:35.483033 6344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:35.482985 6343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:35.483069 6346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:35.483923 6338 server_base.cc:1034] running on GCE node
I20260619 13:07:35.484629 6338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:35.485685 6338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:35.486927 6338 hybrid_clock.cc:648] HybridClock initialized: now 1781874455486866 us; error 60 us; skew 500 ppm
I20260619 13:07:35.487293 6338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.490566 6338 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:35.492188 6338 webserver.cc:460] Webserver started at http://127.0.97.3:39475/ using document root <none> and password file <none>
I20260619 13:07:35.492920 6338 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:35.493067 6338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:35.493391 6338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:35.496423 6338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "39cf53a0b7974dc1b72b1de1f256aaf1"
format_stamp: "Formatted at 2026-06-19 13:07:35 on dist-test-slave-q11v"
server_key: "b7b5a7ae803513ebe1dd090c7eb3c78c"
server_key_iv: "619ea107c33be8cf6c8af6d8cbe6075e"
server_key_version: "encryptionkey@0"
I20260619 13:07:35.497267 6338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "39cf53a0b7974dc1b72b1de1f256aaf1"
format_stamp: "Formatted at 2026-06-19 13:07:35 on dist-test-slave-q11v"
server_key: "b7b5a7ae803513ebe1dd090c7eb3c78c"
server_key_iv: "619ea107c33be8cf6c8af6d8cbe6075e"
server_key_version: "encryptionkey@0"
I20260619 13:07:35.503381 6338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260619 13:07:35.506945 6353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.508594 6338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:35.508726 6338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "39cf53a0b7974dc1b72b1de1f256aaf1"
format_stamp: "Formatted at 2026-06-19 13:07:35 on dist-test-slave-q11v"
server_key: "b7b5a7ae803513ebe1dd090c7eb3c78c"
server_key_iv: "619ea107c33be8cf6c8af6d8cbe6075e"
server_key_version: "encryptionkey@0"
I20260619 13:07:35.508898 6338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:35.530630 6338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:35.534538 6338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:35.534786 6338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:35.535687 6338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:35.537014 6338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:35.537086 6338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.537168 6338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:35.537212 6338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:35.571142 6338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42931
I20260619 13:07:35.571159 6465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42931 every 8 connection(s)
I20260619 13:07:35.572435 6338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:35.578976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6338
I20260619 13:07:35.579169 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:35.579619 388 external_mini_cluster.cc:1428] Setting key 9d9f8d84aa1f39c1cbf723265499eda6
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:35.592679 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.575182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59171 (local address 127.0.97.62:37735)
0619 13:07:35.575380 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.575384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.576506 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 13:07:35.578335 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.578340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.578406 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.578500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.580261 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.581281 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.583318 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.583690 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.583787 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.587322 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.587326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.587330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.587371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.589829 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.590506 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.590509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.590512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.590631 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.591057 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.591060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.591062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.591305 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.591408 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.592254 (+ 846us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.592434 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:07:35.593948 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.575627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37735 (local address 127.0.97.3:59171)
0619 13:07:35.576347 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:35.576384 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:35.578066 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:35.578776 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:35.578785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:35.579440 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:35.580072 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:35.580095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.581469 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.581475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:35.583157 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:35.583169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.583876 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.583881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:35.584087 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.584739 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:35.584761 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:35.587101 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:35.589953 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:35.589960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:35.589973 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:35.590396 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:35.590757 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:35.590761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:35.590764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:35.590954 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:35.591467 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:35.591479 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:35.591915 (+ 436us) client_negotiation.cc:769] Sending connection context
0619 13:07:35.592160 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 13:07:35.592548 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":288,"threads_started":1}
I20260619 13:07:35.595549 6466 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37735
I20260619 13:07:35.595826 6466 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:35.596525 6466 heartbeater.cc:511] Master 127.0.97.62:37735 requested a full tablet report, sending...
I20260619 13:07:35.597939 6009 ts_manager.cc:194] Registered new tserver with Master: 39cf53a0b7974dc1b72b1de1f256aaf1 (127.0.97.3:42931)
I20260619 13:07:35.598800 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59171
I20260619 13:07:35.600606 6199 heartbeater.cc:503] Master 127.0.97.62:37735 was elected leader, sending a full tablet report...
I20260619 13:07:35.607410 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/test-admin.keytab.
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.632815 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 13:07:35.637843 6473 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:35.635466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.0.97.1:44741)
0619 13:07:35.636101 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.636108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.636137 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:07:35.636203 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.636209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.636417 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.636564 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.637425 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35292: BlockingRecv error: recv got EOF from 127.0.0.1:35292 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":340,"threads_started":1}
W20260619 13:07:35.637923 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.639670 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.641012 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:35.656114 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.642723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.0.97.1:44741)
0619 13:07:35.642924 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.642930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.643074 (+ 144us) server_negotiation.cc:407] Connection header received
0619 13:07:35.643221 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.643224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.643284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.643368 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.644251 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.645204 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.646503 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.647048 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.647171 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.650715 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.650727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.650741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.650779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.653530 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.654200 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.654227 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.654229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.654354 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.654670 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.654673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.654675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.655009 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.655244 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.655550 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.655822 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 13:07:35.661659 6473 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:35.660001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.0.97.1:44741)
0619 13:07:35.660215 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.660221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.660417 (+ 196us) server_negotiation.cc:407] Connection header received
0619 13:07:35.660530 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.660534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.660586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.660661 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.661396 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35308: BlockingRecv error: recv got EOF from 127.0.0.1:35308 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 13:07:35.662019 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.663509 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.664811 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:35.678825 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.666137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.0.97.1:44741)
0619 13:07:35.666375 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.666382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.666624 (+ 242us) server_negotiation.cc:407] Connection header received
0619 13:07:35.666806 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.666812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.666875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.666978 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.667872 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.668536 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.669778 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.670202 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.670349 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.673709 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.673715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.673718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.673748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.676124 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.676798 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.676821 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.676825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.676960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.677358 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.677365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.677369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.677719 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.678007 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.678305 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.678524 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 13:07:35.683048 6473 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:35.681538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.97.1:44741)
0619 13:07:35.681750 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.681755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.681916 (+ 161us) server_negotiation.cc:407] Connection header received
0619 13:07:35.682040 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.682043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.682106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.682180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.682840 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35326: BlockingRecv error: recv got EOF from 127.0.0.1:35326 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 13:07:35.683185 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874405339864-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.684947 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:35.686369 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:35.700155 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.687747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.97.1:44741)
0619 13:07:35.687982 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.687987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.688233 (+ 246us) server_negotiation.cc:407] Connection header received
0619 13:07:35.688356 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.688359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.688411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.688489 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.689507 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.690305 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.691518 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.691934 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.692037 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.695483 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.695487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.695491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.695519 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.697868 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.698486 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.698492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.698495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.698626 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.698934 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.698938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.698940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.699199 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.699476 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.699647 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.699846 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 13:07:35.701778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6071
I20260619 13:07:35.713497 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.702559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.0.97.1:44741)
0619 13:07:35.702785 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.702790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.702945 (+ 155us) server_negotiation.cc:407] Connection header received
0619 13:07:35.703108 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.703111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.703163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.703241 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.704120 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.704783 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.706083 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.706620 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.706722 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.708759 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.708763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.708766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.708825 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.711313 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.711853 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.711857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.711859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.711967 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.712320 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.712323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.712325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.712545 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.712731 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.712962 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.713195 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:07:35.714074 6194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:35.795612 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6071
I20260619 13:07:35.811868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6205
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:35.826617 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.812846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.97.2:41117)
0619 13:07:35.813427 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.813435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.813459 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:07:35.813520 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.813526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.813811 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.813943 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.814820 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.815825 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.816974 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.817490 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.817753 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.821342 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.821352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.821366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.821397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.823913 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.824582 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.824590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.824592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.824720 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.825094 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.825101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.825104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.825426 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.825764 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.826062 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.826306 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":345,"threads_started":1}
I20260619 13:07:35.827997 6328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:35.909799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6205
I20260619 13:07:35.926237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6338
Jun 19 13:07:35 dist-test-slave-q11v krb5kdc[5965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:07:35.940917 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:35.927384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.0.97.3:42931)
0619 13:07:35.928045 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:35.928052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:35.928081 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:07:35.928137 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:35.928143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:35.928357 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:35.928482 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:35.929443 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.930498 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.931616 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:35.932148 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:35.932239 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:35.936129 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:35.936139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:35.936153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:35.936191 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:35.938730 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.939299 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.939306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.939308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.939416 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:35.939714 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:35.939717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:35.939719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:35.940026 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:35.940250 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:35.940453 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 13:07:35.940637 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":384,"threads_started":1}
I20260619 13:07:35.942335 6461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:36.026120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6338
I20260619 13:07:36.041747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 5978
I20260619 13:07:36.042924 6040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:36.143983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 5978
2026-06-19T13:07:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3926 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:36 dist-test-slave-q11v krb5kdc[6493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:36 dist-test-slave-q11v krb5kdc[6493](info): set up 2 sockets
Jun 19 13:07:36 dist-test-slave-q11v krb5kdc[6493](info): commencing operation
krb5kdc: starting...
W20260619 13:07:37.250730 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:37 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 13:07:37.456416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37865
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:37.795893 6506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:37.796254 6506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:37.796406 6506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:37.803591 6506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:37.803745 6506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:37.803858 6506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:37.803962 6506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:37.804044 6506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:37.813805 6506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:37.815158 6506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:37.816500 6506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:37.827646 6511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:37.827737 6512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:37.827698 6514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:37.828523 6506 server_base.cc:1034] running on GCE node
I20260619 13:07:37.829286 6506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:37.830678 6506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:37.831894 6506 hybrid_clock.cc:648] HybridClock initialized: now 1781874457831852 us; error 29 us; skew 500 ppm
I20260619 13:07:37.832248 6506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:37 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:37.835505 6506 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 13:07:37.836879 6506 webserver.cc:460] Webserver started at http://127.0.97.62:37441/ using document root <none> and password file <none>
I20260619 13:07:37.837672 6506 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:37.837800 6506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:37.838063 6506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:37.840905 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "2ec55716195d4a65b48d163a1edb36f4"
format_stamp: "Formatted at 2026-06-19 13:07:37 on dist-test-slave-q11v"
server_key: "3eb6173bae49e1acef4decce343e8a7e"
server_key_iv: "6669784feeaec9a232b15abb165e8385"
server_key_version: "encryptionkey@0"
I20260619 13:07:37.841662 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ec55716195d4a65b48d163a1edb36f4"
format_stamp: "Formatted at 2026-06-19 13:07:37 on dist-test-slave-q11v"
server_key: "3eb6173bae49e1acef4decce343e8a7e"
server_key_iv: "6669784feeaec9a232b15abb165e8385"
server_key_version: "encryptionkey@0"
I20260619 13:07:37.846716 6506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:07:37.850135 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:37.851526 6506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:07:37.851660 6506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "2ec55716195d4a65b48d163a1edb36f4"
format_stamp: "Formatted at 2026-06-19 13:07:37 on dist-test-slave-q11v"
server_key: "3eb6173bae49e1acef4decce343e8a7e"
server_key_iv: "6669784feeaec9a232b15abb165e8385"
server_key_version: "encryptionkey@0"
I20260619 13:07:37.851820 6506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:37.874107 6506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:37.878234 6506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:37.878481 6506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:37.896858 6506 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37865
I20260619 13:07:37.896881 6572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37865 every 8 connection(s)
I20260619 13:07:37.898319 6506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:37.902405 6573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:37.906172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6506
I20260619 13:07:37.906400 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:37.907198 388 external_mini_cluster.cc:1428] Setting key 149c3d118463cb86c567c6e41e14a054
I20260619 13:07:37.916258 6573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Bootstrap starting.
I20260619 13:07:37.920132 6573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:37.921469 6573 log.cc:824] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:07:37 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:37.924625 6573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: No bootstrap required, opened a new log
I20260619 13:07:37.930827 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:37.909478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.0.97.62:37865)
0619 13:07:37.910367 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:37.910380 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:37.910424 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:07:37.912048 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:37.912072 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:37.912516 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:37.912868 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:37.914685 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:37.916022 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:37.917756 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:37.918468 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:37.918607 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:37.923568 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:37.923576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:37.923591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:37.923629 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:37.926813 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:37.927634 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:37.927640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:37.927646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:37.927795 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:37.928172 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:37.928175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:37.928177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:37.928670 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:37.928894 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:37.929351 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 13:07:37.929774 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":406,"threads_started":1}
I20260619 13:07:37.933190 6573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } }
I20260619 13:07:37.933696 6573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:37.933789 6573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec55716195d4a65b48d163a1edb36f4, State: Initialized, Role: FOLLOWER
I20260619 13:07:37.934408 6573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } }
I20260619 13:07:37.934640 6573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:37.934772 6573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:37.934937 6573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:37.937278 6573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } }
I20260619 13:07:37.937814 6573 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec55716195d4a65b48d163a1edb36f4; no voters:
I20260619 13:07:37.938378 6573 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:37.938524 6578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:37.939188 6578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec55716195d4a65b48d163a1edb36f4, State: Running, Role: LEADER
I20260619 13:07:37.939791 6578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } }
I20260619 13:07:37.940541 6573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:37.942776 6580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec55716195d4a65b48d163a1edb36f4. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } }
I20260619 13:07:37.943207 6580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:37.943959 6579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } }
I20260619 13:07:37.944535 6579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:37.945034 6585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:37.949918 6585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:37.959424 6585 catalog_manager.cc:1347] Generated new cluster ID: 720167b46e694fa9abdcff7be8bebb3a
I20260619 13:07:37.959576 6585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:38.002265 6585 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:38.003599 6585 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:38.011330 6585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Generated new TSK 0
I20260619 13:07:38.012348 6585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 13:07:38.073227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:38.408041 6599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:38.408396 6599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:38.408583 6599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:38.416218 6599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:38.416425 6599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:38.416642 6599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:38.426826 6599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:38.428196 6599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:38.429539 6599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:38.443755 6604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:38.443791 6605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:38.443722 6607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:38.444828 6599 server_base.cc:1034] running on GCE node
I20260619 13:07:38.445492 6599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:38.446563 6599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:38.447824 6599 hybrid_clock.cc:648] HybridClock initialized: now 1781874458447768 us; error 52 us; skew 500 ppm
I20260619 13:07:38.448220 6599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:38 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:38.451694 6599 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:07:38.453137 6599 webserver.cc:460] Webserver started at http://127.0.97.1:38589/ using document root <none> and password file <none>
I20260619 13:07:38.453982 6599 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:38.454115 6599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:38.454427 6599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:38.457563 6599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "96950338f6c44e0282ba11e5ea2e0289"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "d47c37838634af3c81f97f6ea680be00"
server_key_iv: "b82e1be2cf1126436596f57995ab2809"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.458455 6599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "96950338f6c44e0282ba11e5ea2e0289"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "d47c37838634af3c81f97f6ea680be00"
server_key_iv: "b82e1be2cf1126436596f57995ab2809"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.463622 6599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:38.467060 6614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:38.468341 6599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:07:38.468506 6599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "96950338f6c44e0282ba11e5ea2e0289"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "d47c37838634af3c81f97f6ea680be00"
server_key_iv: "b82e1be2cf1126436596f57995ab2809"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.468709 6599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:38.494879 6599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:38.499271 6599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:38.499583 6599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:38.500583 6599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:38.502027 6599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:38.502099 6599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:38.502243 6599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:38.502312 6599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:38.535166 6599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32951
I20260619 13:07:38.535279 6726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32951 every 8 connection(s)
I20260619 13:07:38.536463 6599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:38.545786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6599
I20260619 13:07:38.546006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:38.546631 388 external_mini_cluster.cc:1428] Setting key fe561da9ac1e8516abd355448caa942a
Jun 19 13:07:38 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:38.558980 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:38.538998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54095 (local address 127.0.97.62:37865)
0619 13:07:38.539465 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:38.539471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:38.540572 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 13:07:38.542548 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:38.542553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:38.542617 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:38.542705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:38.544243 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:38.544994 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:38.546857 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:38.547465 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:38.547598 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:38.551637 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:38.551647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:38.551654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:38.551698 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:38.555442 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:38.556581 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:38.556585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:38.556588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:38.556707 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:38.557217 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:38.557221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:38.557223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:38.557493 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:38.557683 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:38.558550 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 13:07:38.558720 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":218,"threads_started":1}
I20260619 13:07:38.560565 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:38.539593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:54095)
0619 13:07:38.540375 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:38.540441 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:38.542264 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:38.542960 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:38.542973 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:38.543512 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:38.544085 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:38.544103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:38.545156 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:38.545160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:38.546676 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:38.546693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:38.547675 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:38.547701 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:38.547983 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:38.548781 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:38.548804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:38.551369 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:38.555582 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:38.555595 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:38.555619 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:38.556360 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:38.556815 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:38.556822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:38.556828 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:38.557081 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:38.557648 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:38.557659 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:38.558133 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 13:07:38.558450 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 13:07:38.559064 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":280,"threads_started":1}
I20260619 13:07:38.562263 6727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:38.562579 6727 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:38.563305 6727 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:38.565972 6538 ts_manager.cc:194] Registered new tserver with Master: 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951)
I20260619 13:07:38.568166 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54095
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 13:07:38.596140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:38.920431 6733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:38.920689 6733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:38.920853 6733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:38.928453 6733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:38.928560 6733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:38.928726 6733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:38.939563 6733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:38.941185 6733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:38.942858 6733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:38.957733 6739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:38.957818 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:38.957764 6738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:38.958678 6733 server_base.cc:1034] running on GCE node
I20260619 13:07:38.959278 6733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:38.960186 6733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:38.961410 6733 hybrid_clock.cc:648] HybridClock initialized: now 1781874458961355 us; error 48 us; skew 500 ppm
I20260619 13:07:38.961797 6733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:38 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:38.964839 6733 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:07:38.966363 6733 webserver.cc:460] Webserver started at http://127.0.97.2:46239/ using document root <none> and password file <none>
I20260619 13:07:38.967044 6733 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:38.967185 6733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:38.967487 6733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:38.970360 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "a77f5f40817949d4a40cd63d82b75135"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "ac937bf5923bc847ef3cea0c604fe6a2"
server_key_iv: "d4e8575de8ac17c2fc0c3f0fd9b98f97"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.971125 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a77f5f40817949d4a40cd63d82b75135"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "ac937bf5923bc847ef3cea0c604fe6a2"
server_key_iv: "d4e8575de8ac17c2fc0c3f0fd9b98f97"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.976274 6733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:38.979589 6748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:38.980880 6733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:38.981060 6733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "a77f5f40817949d4a40cd63d82b75135"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "ac937bf5923bc847ef3cea0c604fe6a2"
server_key_iv: "d4e8575de8ac17c2fc0c3f0fd9b98f97"
server_key_version: "encryptionkey@0"
I20260619 13:07:38.981261 6733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:38.994442 6733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:38.998816 6733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:38.999122 6733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:39.000088 6733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:39.001376 6733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:39.001480 6733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:39.001667 6733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:39.001745 6733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 13:07:39.034499 6733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44983
I20260619 13:07:39.034575 6860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44983 every 8 connection(s)
I20260619 13:07:39.035787 6733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:39.036201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6733
I20260619 13:07:39.036393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:39.037003 388 external_mini_cluster.cc:1428] Setting key 86b951dfb811e26dc516c0264a65cc88
Jun 19 13:07:39 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:07:39.055979 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:39.038563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58061 (local address 127.0.97.62:37865)
0619 13:07:39.038733 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:39.038737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:39.040066 (+ 1329us) server_negotiation.cc:407] Connection header received
0619 13:07:39.042035 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:39.042041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:39.042121 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:39.042240 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:39.043711 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.044415 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.046021 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.046436 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.046567 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:39.050243 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:39.050247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:39.050251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:39.050280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:39.052906 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:39.053659 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:39.053666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:39.053669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:39.053799 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:39.054237 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:39.054242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:39.054247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:39.054496 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:39.054644 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:39.055376 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 13:07:39.055591 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:07:39.057011 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:39.039189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:58061)
0619 13:07:39.039912 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:39.039959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:39.041745 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:39.042481 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:39.042489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:39.042979 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:39.043544 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:39.043561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.044557 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.044562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:39.045886 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:39.045896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.046650 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.046656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:39.046877 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:39.047551 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:39.047573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:39.050014 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:39.053062 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:39.053070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:39.053087 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:39.053502 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:39.053923 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:39.053930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:39.053935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:39.054122 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:39.054618 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:39.054626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:39.054958 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 13:07:39.055285 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 13:07:39.055829 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260619 13:07:39.058516 6861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:39.058828 6861 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:39.059497 6861 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:39.060935 6538 ts_manager.cc:194] Registered new tserver with Master: a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:39.062186 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:07:39.081869 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:39.409776 6866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:39.410184 6866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:39.410285 6866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:39.417491 6866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:39.417635 6866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:39.417814 6866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:39.427587 6866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:39.428887 6866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:39.430204 6866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:39.443833 6871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:39.443918 6874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:39.443918 6872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:39.444721 6866 server_base.cc:1034] running on GCE node
I20260619 13:07:39.445405 6866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:39.446534 6866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:39.447824 6866 hybrid_clock.cc:648] HybridClock initialized: now 1781874459447771 us; error 78 us; skew 500 ppm
I20260619 13:07:39.448171 6866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:39 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:39.451478 6866 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:07:39.452840 6866 webserver.cc:460] Webserver started at http://127.0.97.3:32943/ using document root <none> and password file <none>
I20260619 13:07:39.453536 6866 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:39.453701 6866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:39.454051 6866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:39.457224 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "fb6246d182494890ad91fb0576ec0233"
format_stamp: "Formatted at 2026-06-19 13:07:39 on dist-test-slave-q11v"
server_key: "42d8731adc0d6c0944b6beaf609dd581"
server_key_iv: "2b84826b3593f219346e12812275229a"
server_key_version: "encryptionkey@0"
I20260619 13:07:39.458067 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb6246d182494890ad91fb0576ec0233"
format_stamp: "Formatted at 2026-06-19 13:07:39 on dist-test-slave-q11v"
server_key: "42d8731adc0d6c0944b6beaf609dd581"
server_key_iv: "2b84826b3593f219346e12812275229a"
server_key_version: "encryptionkey@0"
I20260619 13:07:39.463492 6866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:07:39.466908 6881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:39.468215 6866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:39.468389 6866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "fb6246d182494890ad91fb0576ec0233"
format_stamp: "Formatted at 2026-06-19 13:07:39 on dist-test-slave-q11v"
server_key: "42d8731adc0d6c0944b6beaf609dd581"
server_key_iv: "2b84826b3593f219346e12812275229a"
server_key_version: "encryptionkey@0"
I20260619 13:07:39.468598 6866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:39.515605 6866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:39.520061 6866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:39.520375 6866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:39.521350 6866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:39.522837 6866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:39.522945 6866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:39.523106 6866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:39.523180 6866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:39.557412 6866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41925
I20260619 13:07:39.557469 6993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41925 every 8 connection(s)
I20260619 13:07:39.558849 6866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:39.562950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 6866
I20260619 13:07:39.563115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:39.563542 388 external_mini_cluster.cc:1428] Setting key 68f25930f62746236e9c94854ab7ffab
I20260619 13:07:39.571381 6727 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
Jun 19 13:07:39 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:39.579985 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:39.561385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49511 (local address 127.0.97.62:37865)
0619 13:07:39.561843 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:39.561849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:39.562867 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 13:07:39.565140 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:39.565145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:39.565233 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:39.565356 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:39.567102 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.567859 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.569401 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.570084 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.570207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:39.574366 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:39.574372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:39.574377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:39.574416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:39.576867 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:39.577704 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:39.577709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:39.577713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:39.577834 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:39.578259 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:39.578264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:39.578267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:39.578541 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:39.578707 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:39.579411 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 13:07:39.579621 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260619 13:07:39.580950 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:39.562026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:49511)
0619 13:07:39.562695 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:39.562736 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:39.564850 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:39.565633 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:39.565644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:39.566360 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:39.566920 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:39.566937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.568014 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.568020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:39.569250 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:39.569262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:39.570299 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:39.570306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:39.570658 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:39.571361 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:39.571391 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:39.574154 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:39.577008 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:39.577017 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:39.577036 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:39.577560 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:39.577943 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:39.577949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:39.577953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:39.578137 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:39.578679 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:39.578687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:39.579040 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 13:07:39.579308 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 13:07:39.579837 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":264,"threads_started":1}
I20260619 13:07:39.582230 6994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:39.582502 6994 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:39.583158 6994 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:39.584518 6538 ts_manager.cc:194] Registered new tserver with Master: fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:39.585283 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49511
I20260619 13:07:39.590920 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 13:07:39.637308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38469
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37865,127.0.97.61:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:39.964643 7000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:39.965011 7000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:39.965173 7000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:39.972600 7000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:39.972786 7000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:39.972896 7000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:39.972990 7000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:39.973053 7000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:39.983407 7000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37865,127.0.97.61:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
W20260619 13:07:39.983740 7000 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:37865,127.0.97.61:38469'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 13:07:39.984920 7000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:39.986393 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:39.999181 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:39.999226 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:39.999291 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:40.003372 7000 server_base.cc:1034] running on GCE node
I20260619 13:07:40.004226 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:40.005836 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:40.007083 7000 hybrid_clock.cc:648] HybridClock initialized: now 1781874460006990 us; error 71 us; skew 500 ppm
I20260619 13:07:40.007416 7000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:40.013568 7000 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 13:07:40.015339 7000 webserver.cc:460] Webserver started at http://127.0.97.61:33665/ using document root <none> and password file <none>
I20260619 13:07:40.016147 7000 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:40.016265 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:40.016623 7000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:40.019884 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/instance:
uuid: "1ed7c6ab783e4615811805187ba5c5f2"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "2270474c083db4a89a685542c10c8646"
server_key_iv: "d4737eef12e66ce0debdf5c82d6224a4"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.020710 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal/instance:
uuid: "1ed7c6ab783e4615811805187ba5c5f2"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "2270474c083db4a89a685542c10c8646"
server_key_iv: "d4737eef12e66ce0debdf5c82d6224a4"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.026232 7000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:07:40.029978 7015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:40.031497 7000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:07:40.031677 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "1ed7c6ab783e4615811805187ba5c5f2"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "2270474c083db4a89a685542c10c8646"
server_key_iv: "d4737eef12e66ce0debdf5c82d6224a4"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.031908 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:40.054332 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:40.059194 7000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:40.059479 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:40.064765 6861 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
I20260619 13:07:40.079739 7000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38469
I20260619 13:07:40.079767 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38469 every 8 connection(s)
I20260619 13:07:40.081409 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:07:40.085263 7067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:40.088366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7000
I20260619 13:07:40.093442 7067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:40.113535 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.096381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.97.62:37865)
0619 13:07:40.096916 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.096925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.097384 (+ 459us) server_negotiation.cc:407] Connection header received
0619 13:07:40.097618 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.097626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.097709 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.097822 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.099240 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.100217 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.102102 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.102487 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.102590 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.106400 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.106408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.106414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.106477 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.109709 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.110582 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.110588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.110591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.110756 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.111373 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.111380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.111385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.111919 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.112107 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.112791 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.113096 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":254,"threads_started":1}
I20260619 13:07:40.114672 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.090813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.0.97.61:38469)
0619 13:07:40.091671 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.091684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.091727 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:07:40.093370 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.093394 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.093906 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.094260 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.096048 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.098026 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.099652 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.100713 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.100846 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.105945 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.105958 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.105972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.106004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.109630 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.110590 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.110599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.110603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.110816 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.111373 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.111380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.111384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.112311 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.112554 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.113030 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.113444 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":632,"spinlock_wait_cycles":19328,"thread_start_us":414,"threads_started":1}
I20260619 13:07:40.114693 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.096482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52458)
0619 13:07:40.097236 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.097279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.097472 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.097894 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.097901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.098761 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.099083 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.099095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.100367 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.100370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.101963 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.101978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.102734 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.102743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.102963 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.103638 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.103670 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.106186 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.109846 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.109858 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.109870 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.110447 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.110919 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.110925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.110932 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.111178 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.112094 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.112101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.112322 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.112687 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.113281 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1,"thread_start_us":415,"threads_started":1}
I20260619 13:07:40.116700 7067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:07:40.129302 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.117558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34162)
0619 13:07:40.117771 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.117794 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.117955 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.118279 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.118283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.118633 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.118804 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.118813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.119808 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.119812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.120670 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.120682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.121348 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.121352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.121512 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.122152 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.122169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.124103 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.127277 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.127280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.127283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.127627 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.127950 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.127953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.127955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.128078 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.128558 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.128561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.128704 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.128829 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.129047 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:07:40.129472 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.117506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.0.97.61:38469)
0619 13:07:40.117773 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.117791 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.117882 (+ 91us) server_negotiation.cc:407] Connection header received
0619 13:07:40.118093 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.118097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.118151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.118227 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.118966 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.119669 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.120800 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.121195 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.121289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.124271 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.124276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.124279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.124307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.127141 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.127728 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.127731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.127734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.127857 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.128164 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.128167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.128169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.128439 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.128543 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.128925 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.129162 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":138}
I20260619 13:07:40.134176 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Bootstrap starting.
I20260619 13:07:40.137450 7067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:40.138634 7067 log.cc:824] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:40.141453 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: No bootstrap required, opened a new log
I20260619 13:07:40.152469 7067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } }
I20260619 13:07:40.152810 7067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:40.152894 7067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7c6ab783e4615811805187ba5c5f2, State: Initialized, Role: FOLLOWER
I20260619 13:07:40.153522 7067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } }
I20260619 13:07:40.154440 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } }
I20260619 13:07:40.154788 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:40.155479 7067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:07:40.164458 7085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:07:40.164647 7085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:07:40.166237 7000 master_runner.cc:428] Detected that this master 1ed7c6ab783e4615811805187ba5c5f2 is joining an existing cluster
I20260619 13:07:40.166363 7000 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38469
I20260619 13:07:40.189965 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.170410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.0.97.62:37865)
0619 13:07:40.170608 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.170616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.171192 (+ 576us) server_negotiation.cc:407] Connection header received
0619 13:07:40.171545 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.171551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.171685 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.171786 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.174193 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.175673 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.179057 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.179777 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.179883 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.182411 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.182417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.182422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.182460 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.185997 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.187513 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.187522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.187527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.187678 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.188137 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.188142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.188145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.188509 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.188666 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.189254 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.189534 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:07:40.191092 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.170434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52466)
0619 13:07:40.171086 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.171104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.171323 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.172201 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.172208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.173416 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.173972 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.173990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.176323 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.176332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.178811 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.178832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.180208 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.180214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.180385 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.181073 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.181090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.182175 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.186778 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.186782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.186787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.187371 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.187829 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.187834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.187837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.188013 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.188732 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.188737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.188955 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.190225 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.190589 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":955,"threads_started":1}
I20260619 13:07:40.197217 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.176092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.97.61:38469)
0619 13:07:40.177256 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.177262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.179780 (+ 2518us) server_negotiation.cc:407] Connection header received
0619 13:07:40.179963 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.179968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.180042 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.180141 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.181455 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.182547 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.185246 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.186033 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.186125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.188697 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.188703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.188709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.188740 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.192215 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.194436 (+ 2221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.194441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.194444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.194596 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.194998 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.195002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.195004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.195715 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.195868 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.196604 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.196859 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 13:07:40.198566 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.171076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34172)
0619 13:07:40.179656 (+ 8580us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.179682 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.179835 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.180191 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.180197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.180928 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.181280 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.181294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.182736 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.182741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.185094 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.185110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.186484 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.186489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.186767 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.187381 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.187406 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.188539 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.193765 (+ 5226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.193769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.193774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.194309 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.194726 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.194730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.194732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.194890 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.195959 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.195966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.196307 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.197773 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.198081 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8441,"thread_start_us":8248,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:07:40.215564 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.199581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.0.97.61:38469)
0619 13:07:40.200953 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.200960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.200979 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:07:40.201064 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.201075 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.201223 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.201328 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.202749 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.203879 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.205889 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.206454 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.206589 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.209901 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.209906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.209909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.209939 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.212788 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.213496 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.213502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.213505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.213698 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.214074 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.214079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.214082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.214354 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.214502 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.214956 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.215171 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246}
I20260619 13:07:40.215653 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.199201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34176)
0619 13:07:40.200418 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.200466 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.200630 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.201406 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.201415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.202306 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.202589 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.202602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.204037 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.204043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.205731 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.205748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.206633 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.206638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.206846 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.207515 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.207533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.209709 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.212930 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.212944 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.212952 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.213371 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.213803 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.213818 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.213822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.213960 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.214484 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.214493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.214671 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.214859 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.215195 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
I20260619 13:07:40.217909 6538 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38469
}
attrs {
promote: true
}
}
dest_uuid: "2ec55716195d4a65b48d163a1edb36f4"
cas_config_opid_index: -1
I20260619 13:07:40.218861 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } }
I20260619 13:07:40.222606 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.226955 7097 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38469
I20260619 13:07:40.227421 7042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:40.227406 7097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.227620 7097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 13:07:40.229523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44491
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:07:40.231580 7042 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 13:07:40.233476 7000 master.cc:537] Master@127.0.97.61:38469 shutting down...
I20260619 13:07:40.234503 7098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:07:40.239221 6524 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 13:07:40.247164 6524 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 -> Peer 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61:38469): Couldn't send request to peer 1ed7c6ab783e4615811805187ba5c5f2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 13:07:40.247260 7000 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:40.247699 7000 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:40.247903 7000 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: stopping tablet replica
I20260619 13:07:40.263960 7000 master.cc:552] Master@127.0.97.61:38469 shutdown complete.
I20260619 13:07:40.264075 7000 master_runner.cc:305] Clearing existing system tablet
I20260619 13:07:40.264467 7000 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:40.264575 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:40.267097 7000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 13:07:40.268620 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:40.268931 7000 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:40.269232 7000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 13:07:40.269325 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "1ed7c6ab783e4615811805187ba5c5f2"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "2270474c083db4a89a685542c10c8646"
server_key_iv: "d4737eef12e66ce0debdf5c82d6224a4"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.273087 7000 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:40.277827 7000 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:07:40.278021 7000 log.cc:1196] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 13:07:40.278448 7000 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Deleting consensus metadata
I20260619 13:07:40.278697 7000 master_runner.cc:315] Copying system tablet from 127.0.97.62:37865
I20260619 13:07:40.281798 7000 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37865
I20260619 13:07:40.297318 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.282674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.0.97.62:37865)
0619 13:07:40.282913 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.282922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.286124 (+ 3202us) server_negotiation.cc:407] Connection header received
0619 13:07:40.286334 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.286339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.286428 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.286537 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.287385 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.288322 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.289651 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.290091 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.290233 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.292091 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.292096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.292099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.292130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.294729 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.295381 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.295387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.295391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.295520 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.295879 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.295887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.295890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.296159 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.296312 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.296716 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.296929 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 13:07:40.297472 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.282536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52470)
0619 13:07:40.285985 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.286010 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.286199 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.286564 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.286569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.287015 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.287227 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.287236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.288453 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.288460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.289473 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.289485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.290246 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.290251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.290445 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.291178 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.291195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.291878 (+ 683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.294874 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.294880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.294884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.295249 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.295635 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.295640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.295644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.295774 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.296273 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.296278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.296474 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.296618 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.296932 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3317,"thread_start_us":3230,"threads_started":1}
I20260619 13:07:40.298591 6558 tablet_copy_service.cc:140] P 2ec55716195d4a65b48d163a1edb36f4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1ed7c6ab783e4615811805187ba5c5f2 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52470)
I20260619 13:07:40.298858 6558 tablet_copy_service.cc:161] P 2ec55716195d4a65b48d163a1edb36f4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1ed7c6ab783e4615811805187ba5c5f2 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52470: session id = 1ed7c6ab783e4615811805187ba5c5f2-00000000000000000000000000000000
I20260619 13:07:40.302008 6558 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:07:40.303983 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:07:40.308691 7000 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: tablet copy: Starting download of 0 data blocks...
I20260619 13:07:40.309022 7000 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: tablet copy: Starting download of 1 WAL segments...
I20260619 13:07:40.311738 7000 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:07:40.314985 6558 tablet_copy_service.cc:342] P 2ec55716195d4a65b48d163a1edb36f4: Request end of tablet copy session 1ed7c6ab783e4615811805187ba5c5f2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52470
I20260619 13:07:40.315186 6558 tablet_copy_service.cc:433] P 2ec55716195d4a65b48d163a1edb36f4: ending tablet copy session 1ed7c6ab783e4615811805187ba5c5f2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1ed7c6ab783e4615811805187ba5c5f2
I20260619 13:07:40.326987 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:40.329905 7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:40.335171 7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:40.340234 7000 server_base.cc:1034] running on GCE node
W20260619 13:07:40.340579 7113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:40.341100 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:40.345856 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:40.347106 7000 hybrid_clock.cc:648] HybridClock initialized: now 1781874460347020 us; error 60 us; skew 500 ppm
I20260619 13:07:40.347375 7000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:40.349686 7000 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 13:07:40.354581 7000 webserver.cc:460] Webserver started at http://127.0.97.61:44847/ using document root <none> and password file <none>
I20260619 13:07:40.354960 7000 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:40.355041 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:40.358312 7000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:07:40.360270 7120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:40.360720 7000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 13:07:40.360867 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "1ed7c6ab783e4615811805187ba5c5f2"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "2270474c083db4a89a685542c10c8646"
server_key_iv: "d4737eef12e66ce0debdf5c82d6224a4"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.361028 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:40.377079 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:40.381264 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:40.413564 7000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38469
I20260619 13:07:40.413626 7171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38469 every 8 connection(s)
I20260619 13:07:40.414608 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:07:40.420342 7172 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:07:40.424120 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Bootstrap starting.
I20260619 13:07:40.443082 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:07:40.443765 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: Bootstrap complete.
I20260619 13:07:40.445115 7172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } }
I20260619 13:07:40.445412 7172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ed7c6ab783e4615811805187ba5c5f2, State: Initialized, Role: LEARNER
I20260619 13:07:40.445685 7172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } }
I20260619 13:07:40.446548 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.446879 7172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:40.447003 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:07:40.455464 7186 catalog_manager.cc:1259] Loaded cluster ID: 720167b46e694fa9abdcff7be8bebb3a
I20260619 13:07:40.455585 7186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: loading cluster ID for follower catalog manager: success
I20260619 13:07:40.459216 7186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: acquiring CA information for follower catalog manager: success
I20260619 13:07:40.461911 7186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:07:40.587823 6994 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
W20260619 13:07:40.632849 7100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:40.633216 7100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:40.633373 7100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:40.641299 7100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:40.641448 7100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:40.641525 7100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:40.641649 7100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:40.641750 7100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:40.652630 7100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:40.654178 7100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:40.655745 7100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:40.668948 7191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:40.669121 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:40.669685 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:40.670271 7100 server_base.cc:1034] running on GCE node
I20260619 13:07:40.671149 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:40.672735 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:40.673933 7100 hybrid_clock.cc:648] HybridClock initialized: now 1781874460673869 us; error 59 us; skew 500 ppm
I20260619 13:07:40.674244 7100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:40.677259 7100 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 13:07:40.678875 7100 webserver.cc:460] Webserver started at http://127.0.97.60:33331/ using document root <none> and password file <none>
I20260619 13:07:40.679527 7100 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:40.679625 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:40.679888 7100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:40.682955 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/instance:
uuid: "eb4fcd5fc7594f52b70c699e175048ad"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "3c9c57af54a8f173609456fc9300f53c"
server_key_iv: "f9a6bab06784d5ac6c407d9848405116"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.683694 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal/instance:
uuid: "eb4fcd5fc7594f52b70c699e175048ad"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "3c9c57af54a8f173609456fc9300f53c"
server_key_iv: "f9a6bab06784d5ac6c407d9848405116"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.689409 7100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:07:40.692953 7201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:40.694427 7100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:40.694574 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "eb4fcd5fc7594f52b70c699e175048ad"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "3c9c57af54a8f173609456fc9300f53c"
server_key_iv: "f9a6bab06784d5ac6c407d9848405116"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.694741 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:40.706534 7100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:40.710686 7100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:40.710950 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:40.730381 7100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44491
I20260619 13:07:40.730414 7252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44491 every 8 connection(s)
I20260619 13:07:40.731773 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:07:40.735589 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:40.739570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7100
I20260619 13:07:40.744335 7253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:40.766089 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.748383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.0.97.62:37865)
0619 13:07:40.748584 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.748591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.749749 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 13:07:40.750018 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.750025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.750099 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.750207 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.751567 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.752372 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.754150 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.754556 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.754658 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.759247 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.759252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.759255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.759308 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.762468 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.763404 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.763410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.763414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.763602 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.764161 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.764166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.764169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.764525 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.764692 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.765490 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.765759 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:07:40.767213 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.748329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52486)
0619 13:07:40.749151 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.749208 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.749839 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.750272 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.750279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.751098 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.751414 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.751427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.752510 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.752514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.754003 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.754017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.754793 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.754800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.755031 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.755774 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.755810 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.759029 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.762605 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.762618 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.762632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.763247 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.763767 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.763772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.763775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.763987 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.764680 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.764687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.765005 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.765344 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.766282 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":354,"spinlock_wait_cycles":18176,"thread_start_us":413,"threads_started":1}
I20260619 13:07:40.767253 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.742111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.97.60:44491)
0619 13:07:40.742926 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.742941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.742993 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:07:40.745325 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.745361 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.745977 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.746458 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.748209 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.750544 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.752167 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.752991 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.753121 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.758565 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.758575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.758590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.758663 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.762435 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.763408 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.763416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.763419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.763672 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.764148 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.764154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.764156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.764921 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.765147 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.765567 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.765970 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":577,"thread_start_us":332,"threads_started":1}
I20260619 13:07:40.768778 7253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:07:40.782194 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.769459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34188)
0619 13:07:40.769654 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.769674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.769801 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.770464 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.770467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.770787 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.770920 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.770927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.772192 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.772196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.773271 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.773290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.774042 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.774046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.774190 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.774739 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.774755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.776705 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.780068 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.780072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.780075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.780460 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.780841 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.780845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.780847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.780969 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.781418 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.781421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.781539 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.781711 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.781938 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:07:40.782263 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.769600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.97.61:38469)
0619 13:07:40.770163 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.770168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.770207 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:40.770268 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.770273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.770340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.770427 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.771040 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.772021 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.773410 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.773923 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.774023 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.776920 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.776926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.776929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.776964 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.779932 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.780575 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.780580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.780582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.780745 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.781052 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.781055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.781057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.781320 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.781442 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.781835 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.781999 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
I20260619 13:07:40.783376 7253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:07:40.796589 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.789946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.97.61:38469)
0619 13:07:40.790135 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.790142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.790324 (+ 182us) server_negotiation.cc:407] Connection header received
0619 13:07:40.790547 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.790552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.790720 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.790823 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:40.791340 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.792098 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.793881 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.795585 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.795703 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.795912 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.796156 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.796305 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:07:40.796600 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.789708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34190)
0619 13:07:40.790218 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.790239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.790387 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.790888 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.790895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.790928 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:40.791156 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.791171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.792226 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.792234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.793715 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.793728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.795747 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.795753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.795960 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.796025 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.796106 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.796228 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":240,"threads_started":1}
I20260619 13:07:40.797106 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.784157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.97.60:44491)
0619 13:07:40.784308 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.784312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.784465 (+ 153us) server_negotiation.cc:407] Connection header received
0619 13:07:40.784591 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.784594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.784646 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.784789 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.785474 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.786188 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.787404 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.787797 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.787941 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.791422 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.791428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.791434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.791468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.794377 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.794965 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.794972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.794976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.795093 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.795454 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.795462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.795465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.795787 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.795927 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.796345 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.796863 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":50}
I20260619 13:07:40.797305 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.784161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.0.1:54504)
0619 13:07:40.784396 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.784410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.784495 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.784798 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.784801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.785177 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.785322 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.785330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.786347 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.786351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.787250 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.787266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.787982 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.787987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.788179 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.788815 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.788832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.791275 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.794502 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.794506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.794509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.794852 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.795193 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.795200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.795205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.795358 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.795919 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.795925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.796064 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.796242 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.796524 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 13:07:40.797899 7147 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 13:07:40.798481 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec55716195d4a65b48d163a1edb36f4. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.798732 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:07:40.802224 7253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Bootstrap starting.
I20260619 13:07:40.805311 7253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:40.806381 7253 log.cc:824] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:40.808988 7253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: No bootstrap required, opened a new log
I20260619 13:07:40.816403 7253 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } }
I20260619 13:07:40.816706 7253 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:40.816851 7253 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4fcd5fc7594f52b70c699e175048ad, State: Initialized, Role: FOLLOWER
I20260619 13:07:40.817409 7253 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } }
I20260619 13:07:40.818310 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } } }
I20260619 13:07:40.818642 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:40.819341 7253 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:07:40.828680 7273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:07:40.828850 7273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:07:40.833061 7100 master_runner.cc:428] Detected that this master eb4fcd5fc7594f52b70c699e175048ad is joining an existing cluster
I20260619 13:07:40.833184 7100 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44491
I20260619 13:07:40.857172 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.838144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.97.61:38469)
0619 13:07:40.838360 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.838366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.839285 (+ 919us) server_negotiation.cc:407] Connection header received
0619 13:07:40.839638 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.839644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.839730 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.839840 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.841690 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.842808 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.845720 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.846380 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.846483 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.849329 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.849336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.849341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.849384 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.853480 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.854371 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.854380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.854387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.854575 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.855147 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.855152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.855156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.855539 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.855710 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.856477 (+ 767us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.856740 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:07:40.858520 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.838110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34198)
0619 13:07:40.839125 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.839153 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.839382 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.840386 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.840391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.841107 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.841482 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.841495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.842995 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.843000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.845520 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.845536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.846824 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.846831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.847052 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.847859 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.847888 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.849055 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.853650 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.853655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.853660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.854225 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.854717 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.854724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.854730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.854983 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.855802 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.855808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.856095 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.857469 (+ 1374us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.857859 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":26,"thread_start_us":638,"threads_started":1}
I20260619 13:07:40.861923 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.838036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.97.62:37865)
0619 13:07:40.838266 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.838274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.843304 (+ 5030us) server_negotiation.cc:407] Connection header received
0619 13:07:40.843507 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.843514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.843610 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.843720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.845730 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.846901 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.849360 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.850126 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.850252 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.853491 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.853500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.853506 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.853551 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.857849 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.859234 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.859240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.859244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.859412 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.859944 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.859953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.859957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.860356 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.860483 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.861167 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.861463 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 13:07:40.862855 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.840131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.0.97.60:44491)
0619 13:07:40.840309 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.840314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.845020 (+ 4706us) server_negotiation.cc:407] Connection header received
0619 13:07:40.845289 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.845300 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.845380 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.845469 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.846940 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.848232 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.850380 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.851180 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.851308 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.854638 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.854644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.854648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.854681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.859073 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.859830 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.859837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.859845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.860020 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.860397 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.860406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.860409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.860765 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.860892 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.861424 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.861669 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":53}
I20260619 13:07:40.863687 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.838568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52502)
0619 13:07:40.843176 (+ 4608us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.843205 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.843364 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.843761 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.843765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.845076 (+ 1311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.845519 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.845532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.847195 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.847202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.849140 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.849155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.850685 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.850692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.850888 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.851645 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.851675 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.853242 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.858015 (+ 4773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.858026 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.858033 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.859054 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.859576 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.859580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.859584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.859787 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.860581 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.860585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.860831 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.860996 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.861375 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4423,"mutex_wait_us":12,"spinlock_wait_cycles":30592,"thread_start_us":4246,"threads_started":1}
I20260619 13:07:40.863654 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.840150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.0.1:54506)
0619 13:07:40.844918 (+ 4768us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.844944 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.845130 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.845716 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.845720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.846397 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.846764 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.846776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.848499 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.848504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.850232 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.850251 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.851383 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.851392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.851715 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.852379 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.852405 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.854126 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.859216 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.859221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.859225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.859710 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.860151 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.860155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.860159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.860295 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.860960 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.860967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.861121 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.862072 (+ 951us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.862530 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4606,"thread_start_us":3325,"threads_started":1}
I20260619 13:07:40.873695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6599
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 13:07:40 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 13:07:40.885766 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.869182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.0.1:54516)
0619 13:07:40.869424 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:40.869440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:40.869530 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:40.870105 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:40.870118 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:40.870586 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:40.870791 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.870803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.871979 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.871984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.873342 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:40.873359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.874432 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.874441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:40.874615 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.875353 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:40.875376 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:40.878548 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:40.882688 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.882694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.882697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.883286 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.883722 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:40.883728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:40.883732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:40.883900 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:40.884650 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:40.884656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:40.884825 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 13:07:40.885002 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 13:07:40.885300 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 13:07:40.886822 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.869299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.97.60:44491)
0619 13:07:40.869759 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.869763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.869780 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:40.869851 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.869854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.869955 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.870033 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.870945 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.871743 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.873503 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.874247 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.874946 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.878758 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.878765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.878770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.878808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.882535 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.883426 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.883432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.883437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.883576 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.884028 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.884034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.884037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.884421 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.886084 (+ 1663us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.886240 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.886499 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260619 13:07:40.888746 6538 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44491: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44491
}
attrs {
promote: true
}
}
dest_uuid: "2ec55716195d4a65b48d163a1edb36f4"
cas_config_opid_index: 5
I20260619 13:07:40.889904 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:40.891747 7259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.898496 7228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:40.902145 7228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:07:40.902413 7147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:07:40.902801 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:40.874711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.97.1:32951)
0619 13:07:40.875713 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:40.875720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:40.875743 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:07:40.875794 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:40.875800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:40.876013 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:40.876183 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:40.877998 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.879097 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.880531 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:40.881094 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:40.881238 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:40.886460 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:40.886472 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:40.886488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:40.886525 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:40.897358 (+ 10833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.899053 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.899066 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.899071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.899261 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:40.899794 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:40.899799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:40.899802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:40.900312 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:40.900610 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:40.901882 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0619 13:07:40.902189 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":698,"threads_started":1}
I20260619 13:07:40.903541 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:40.904589 6722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:40.905212 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 13:07:40.910353 7259 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44491
I20260619 13:07:40.910724 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.911007 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:40.911008 7100 master.cc:537] Master@127.0.97.60:44491 shutting down...
I20260619 13:07:40.924721 7228 raft_consensus.cc:858] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:40.934980 7147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
W20260619 13:07:40.924916 7228 raft_consensus.cc:1565] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:40.942378 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.942692 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:07:40.944063 7283 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: attempting to promote NON_VOTER 1ed7c6ab783e4615811805187ba5c5f2 to VOTER
I20260619 13:07:40.944810 7283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:40.947333 7147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 LEARNER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 13:07:40.947908 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 13:07:40.950189 7283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.954830 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.955216 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:40.955133 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.955582 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.955803 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:40.958616 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:40.958874 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 13:07:40.970487 6525 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 -> Peer eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60:44491): Couldn't send request to peer eb4fcd5fc7594f52b70c699e175048ad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 13:07:40.970299 7228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.976058 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec55716195d4a65b48d163a1edb36f4. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.976349 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:07:40.975687 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.976531 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: true } } }
I20260619 13:07:40.977034 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:07:40.977149 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:07:40.978955 7100 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 13:07:40.979370 7100 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 13:07:40.979569 7100 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: stopping tablet replica
I20260619 13:07:40.986546 7100 master.cc:552] Master@127.0.97.60:44491 shutdown complete.
I20260619 13:07:40.986680 7100 master_runner.cc:305] Clearing existing system tablet
I20260619 13:07:40.987028 7100 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:40.987154 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:40.989578 7100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 13:07:40.991153 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:40.991482 7100 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:40.991761 7100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 13:07:40.991865 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "eb4fcd5fc7594f52b70c699e175048ad"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "3c9c57af54a8f173609456fc9300f53c"
server_key_iv: "f9a6bab06784d5ac6c407d9848405116"
server_key_version: "encryptionkey@0"
I20260619 13:07:40.996250 7100 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:41.001928 7100 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:07:41.002118 7100 log.cc:1196] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 13:07:41.002631 7100 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Deleting consensus metadata
I20260619 13:07:41.002879 7100 master_runner.cc:315] Copying system tablet from 127.0.97.62:37865
I20260619 13:07:41.005487 7100 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37865
I20260619 13:07:41.019099 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.006366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.97.62:37865)
0619 13:07:41.006651 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.006660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.007189 (+ 529us) server_negotiation.cc:407] Connection header received
0619 13:07:41.007425 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.007435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.007498 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.007611 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:41.008559 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.009282 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.010690 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.011140 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.011276 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.013436 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:41.013450 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:41.013453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:41.013485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:41.016350 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.017032 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.017037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.017041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.017178 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.017567 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.017574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.017577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.017922 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:41.018082 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.018501 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.018740 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 13:07:41.019536 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.006297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52514)
0619 13:07:41.007057 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:41.007083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:41.007263 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:41.007626 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:41.007631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:41.008039 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:41.008405 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.008415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.009439 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.009445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.010532 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.010544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.011310 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.011315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.011497 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.012196 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:41.012215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:41.013228 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:41.016511 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.016516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.016521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.016904 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.017305 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.017311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.017315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.017460 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.018043 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:41.018048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:41.018249 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 13:07:41.018395 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 13:07:41.018738 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":489,"threads_started":1}
I20260619 13:07:41.020141 6558 tablet_copy_service.cc:140] P 2ec55716195d4a65b48d163a1edb36f4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb4fcd5fc7594f52b70c699e175048ad ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52514)
I20260619 13:07:41.020321 6558 tablet_copy_service.cc:161] P 2ec55716195d4a65b48d163a1edb36f4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb4fcd5fc7594f52b70c699e175048ad at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52514: session id = eb4fcd5fc7594f52b70c699e175048ad-00000000000000000000000000000000
I20260619 13:07:41.023134 6558 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:07:41.025127 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:07:41.030148 7100 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: tablet copy: Starting download of 0 data blocks...
I20260619 13:07:41.030438 7100 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: tablet copy: Starting download of 1 WAL segments...
I20260619 13:07:41.032714 7100 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:07:41.037533 6558 tablet_copy_service.cc:342] P 2ec55716195d4a65b48d163a1edb36f4: Request end of tablet copy session eb4fcd5fc7594f52b70c699e175048ad-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52514
I20260619 13:07:41.037727 6558 tablet_copy_service.cc:433] P 2ec55716195d4a65b48d163a1edb36f4: ending tablet copy session eb4fcd5fc7594f52b70c699e175048ad-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb4fcd5fc7594f52b70c699e175048ad
I20260619 13:07:41.049324 7100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:41.052286 7304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:41.061944 7305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:41.064803 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:41.065344 7100 server_base.cc:1034] running on GCE node
I20260619 13:07:41.065728 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:41.066452 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:41.067710 7100 hybrid_clock.cc:648] HybridClock initialized: now 1781874461067671 us; error 29 us; skew 500 ppm
I20260619 13:07:41.067992 7100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:41.070052 7100 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 13:07:41.071305 7100 webserver.cc:460] Webserver started at http://127.0.97.60:33187/ using document root <none> and password file <none>
I20260619 13:07:41.071601 7100 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:41.071710 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:41.074491 7100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:07:41.076282 7314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:41.076751 7100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 13:07:41.076884 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "eb4fcd5fc7594f52b70c699e175048ad"
format_stamp: "Formatted at 2026-06-19 13:07:40 on dist-test-slave-q11v"
server_key: "3c9c57af54a8f173609456fc9300f53c"
server_key_iv: "f9a6bab06784d5ac6c407d9848405116"
server_key_version: "encryptionkey@0"
I20260619 13:07:41.077090 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:41.087456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6599
I20260619 13:07:41.105000 7100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:41.108505 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:41.111171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:32951
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37865
--webserver_port=38589
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:07:41.139639 7100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44491
I20260619 13:07:41.139679 7366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44491 every 8 connection(s)
I20260619 13:07:41.140861 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:07:41.148173 7367 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:07:41.153903 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Bootstrap starting.
I20260619 13:07:41.172134 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:07:41.172729 7367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Bootstrap complete.
I20260619 13:07:41.173686 7367 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:41.173926 7367 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb4fcd5fc7594f52b70c699e175048ad, State: Initialized, Role: LEARNER
I20260619 13:07:41.174176 7367 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } }
I20260619 13:07:41.177873 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:41.178313 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: LEARNER
I20260619 13:07:41.178313 7367 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:41.199715 7381 catalog_manager.cc:1259] Loaded cluster ID: 720167b46e694fa9abdcff7be8bebb3a
I20260619 13:07:41.199841 7381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: loading cluster ID for follower catalog manager: success
I20260619 13:07:41.202316 7381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: acquiring CA information for follower catalog manager: success
I20260619 13:07:41.204046 7381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:07:41.481451 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:41.481767 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:41.481946 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:41.489912 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:41.490020 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:41.490214 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:41.499899 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:32951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38589
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:41.501291 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:41.502660 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:41.516291 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:41.516258 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:41.516419 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:41.517477 7333 server_base.cc:1034] running on GCE node
I20260619 13:07:41.518169 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:41.519060 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:41.520272 7333 hybrid_clock.cc:648] HybridClock initialized: now 1781874461520205 us; error 60 us; skew 500 ppm
I20260619 13:07:41.520592 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:41.523794 7333 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:07:41.525707 7333 webserver.cc:460] Webserver started at http://127.0.97.1:38589/ using document root <none> and password file <none>
I20260619 13:07:41.526425 7333 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:41.526583 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:41.532577 7333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:07:41.535676 7398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:41.537315 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:41.537734 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.530639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.0.97.60:44491)
0619 13:07:41.531741 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.531746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.531762 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:07:41.531821 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.531824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.531895 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.531988 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:41.532443 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.533426 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.535538 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.536749 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.536860 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.537020 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.537288 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.537405 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":815,"threads_started":1}
I20260619 13:07:41.537779 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.530536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.0.1:54520)
0619 13:07:41.531004 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:41.531031 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:41.531233 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:41.532049 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:41.532055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:41.532086 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:41.532296 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.532308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.533577 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.533583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.535374 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.535396 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.536941 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.536948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.537128 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.537162 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:07:41.537268 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:07:41.537425 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260619 13:07:41.537480 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "96950338f6c44e0282ba11e5ea2e0289"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "d47c37838634af3c81f97f6ea680be00"
server_key_iv: "b82e1be2cf1126436596f57995ab2809"
server_key_version: "encryptionkey@0"
I20260619 13:07:41.538539 7342 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 13:07:41.538671 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:41.539017 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec55716195d4a65b48d163a1edb36f4. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:41.539276 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: LEARNER
I20260619 13:07:41.554664 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:41.559444 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:41.559727 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:41.560707 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:41.562198 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:41.562281 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:41.562427 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:41.562531 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:41.597419 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32951
I20260619 13:07:41.597474 7510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32951 every 8 connection(s)
I20260619 13:07:41.598878 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:41.608413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7333
I20260619 13:07:41.608783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6733
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 13:07:41 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 13:07:41.642843 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54429 (local address 127.0.97.61:38469)
0619 13:07:41.621252 (+ 18617us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.621260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.621287 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:07:41.621343 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.621347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.621437 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.621544 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:41.623823 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.624992 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628116 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.628782 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.634617 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:41.634624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:41.634630 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:41.634667 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:41.638775 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.639708 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.639728 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.639733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.639870 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640527 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.640535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.640538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.640934 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:41.641056 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.642123 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.642413 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18479,"thread_start_us":18837,"threads_started":1}
I20260619 13:07:41.642987 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58605 (local address 127.0.97.60:44491)
0619 13:07:41.602302 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.602308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.620808 (+ 18500us) server_negotiation.cc:407] Connection header received
0619 13:07:41.620962 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.620969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.621060 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.621149 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:41.624281 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.625282 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628131 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.628778 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628859 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.635462 (+ 6603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:41.635468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:41.635473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:41.635511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:41.639866 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640752 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.640757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.640760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.640903 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.641361 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.641366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.641369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.641720 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:41.641850 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.642413 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.642663 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:07:41.643527 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43879 (local address 127.0.97.62:37865)
0619 13:07:41.615300 (+ 12616us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.615309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.620110 (+ 4801us) server_negotiation.cc:407] Connection header received
0619 13:07:41.620314 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.620323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.620424 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.620558 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:41.623830 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.625045 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628131 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.628760 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.628858 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.635464 (+ 6606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:41.635470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:41.635474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:41.635511 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:41.639485 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640441 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.640448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.640452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.640588 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.641257 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.641262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.641266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.641690 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:41.641808 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.642698 (+ 890us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.642929 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12435,"thread_start_us":12316,"threads_started":1}
I20260619 13:07:41.645295 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.97.1:58605)
0619 13:07:41.620685 (+ 18006us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:41.620700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:41.620844 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:41.621334 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:41.621338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:41.623825 (+ 2487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:41.624100 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.624111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.625463 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.625468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.627976 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.627986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.629199 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.629209 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.629568 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.631194 (+ 1626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:41.631218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:41.635246 (+ 4028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:41.640007 (+ 4761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.640015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.640625 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.641048 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.641052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.641056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.641238 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.641924 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:41.641928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:41.642096 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 13:07:41.643139 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0619 13:07:41.643853 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17879,"mutex_wait_us":986,"spinlock_wait_cycles":10240,"thread_start_us":12230,"threads_started":1}
I20260619 13:07:41.645321 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.97.1:54429)
0619 13:07:41.615750 (+ 13071us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:41.615802 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:41.619076 (+ 3274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:41.621784 (+ 2708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:41.621788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:41.622843 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:41.623598 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.623627 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.625219 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.625223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.627958 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.627970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.629247 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.629253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.629559 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.630510 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:41.630540 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:41.634320 (+ 3780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:41.638957 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.638967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.638990 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.639563 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.640007 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.640015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.640387 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.641136 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:41.641145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:41.641670 (+ 525us) client_negotiation.cc:769] Sending connection context
0619 13:07:41.643137 (+ 1467us) client_negotiation.cc:240] Negotiation successful
0619 13:07:41.643759 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12760,"mutex_wait_us":414,"spinlock_wait_cycles":57472,"thread_start_us":17612,"threads_started":1}
I20260619 13:07:41.645350 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.602982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:43879)
0619 13:07:41.620007 (+ 17025us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:41.620032 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:41.620177 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:41.621106 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:41.621122 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:41.622493 (+ 1371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:41.623595 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.623630 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.625217 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.625222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.627973 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:41.627986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.629264 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.629270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:41.629572 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.631112 (+ 1540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:41.631148 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:41.635246 (+ 4098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:41.639642 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.639646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.639649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.640299 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.640730 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:41.640734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:41.640737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:41.640891 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:41.641887 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:41.641891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:41.642040 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 13:07:41.643717 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0619 13:07:41.643989 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16859,"mutex_wait_us":492,"thread_start_us":19086,"threads_started":1}
I20260619 13:07:41.646320 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:41.610072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.97.2:44983)
0619 13:07:41.623736 (+ 13664us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:41.623745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:41.623775 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:07:41.623838 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:41.623845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:41.624091 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:41.624252 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:41.626442 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.627963 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.629784 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:41.630534 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:41.630664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:41.636778 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:41.636812 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:41.636832 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:41.636882 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:41.640746 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.641768 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.641777 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.641780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.641943 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:41.642417 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:41.642423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:41.642428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:41.642831 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:41.644817 (+ 1986us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:41.644974 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 13:07:41.645220 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13485,"thread_start_us":13338,"threads_started":1}
I20260619 13:07:41.647049 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:41.647372 7512 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:41.647892 6856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:41.648172 7512 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:41.649017 7513 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38469
I20260619 13:07:41.649161 7513 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:41.649628 7513 heartbeater.cc:511] Master 127.0.97.61:38469 requested a full tablet report, sending...
I20260619 13:07:41.650019 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44491
I20260619 13:07:41.650022 6538 ts_manager.cc:194] Re-registered known tserver with Master: 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951)
I20260619 13:07:41.650131 7511 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:41.650488 7511 heartbeater.cc:511] Master 127.0.97.60:44491 requested a full tablet report, sending...
I20260619 13:07:41.651149 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43879
I20260619 13:07:41.652213 7331 ts_manager.cc:194] Registered new tserver with Master: 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951)
I20260619 13:07:41.651849 7137 ts_manager.cc:194] Registered new tserver with Master: 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951)
I20260619 13:07:41.733711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6733
I20260619 13:07:41.751220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44983
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37865
--webserver_port=46239
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:42.090415 7523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:42.090740 7523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:42.090911 7523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:42.098261 7523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:42.098407 7523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:42.098588 7523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:42.108065 7523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46239
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:42.109375 7523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:42.110742 7523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:42.123893 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:42.123857 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:42.123829 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:42.124722 7523 server_base.cc:1034] running on GCE node
I20260619 13:07:42.125427 7523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:42.126490 7523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:42.127741 7523 hybrid_clock.cc:648] HybridClock initialized: now 1781874462127690 us; error 53 us; skew 500 ppm
I20260619 13:07:42.128122 7523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:42.131407 7523 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:07:42.133004 7523 webserver.cc:460] Webserver started at http://127.0.97.2:46239/ using document root <none> and password file <none>
I20260619 13:07:42.133795 7523 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:42.133963 7523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:42.139847 7523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:07:42.146922 7538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.148172 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: true } } }
I20260619 13:07:42.148550 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:42.149098 7523 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.003s
I20260619 13:07:42.150449 7540 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: attempting to promote NON_VOTER eb4fcd5fc7594f52b70c699e175048ad to VOTER
I20260619 13:07:42.149668 7523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "a77f5f40817949d4a40cd63d82b75135"
format_stamp: "Formatted at 2026-06-19 13:07:38 on dist-test-slave-q11v"
server_key: "ac937bf5923bc847ef3cea0c604fe6a2"
server_key_iv: "d4e8575de8ac17c2fc0c3f0fd9b98f97"
server_key_version: "encryptionkey@0"
I20260619 13:07:42.151088 7523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:42.151194 7540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:42.153405 7342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 LEARNER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:07:42.153661 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Refusing update from remote peer 2ec55716195d4a65b48d163a1edb36f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:07:42.154124 7540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:07:42.154527 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:07:42.158017 7283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.161871 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.163935 7341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.165655 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.166028 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:42.166353 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.166664 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:42.166918 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.167188 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:42.168059 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.168540 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:42.169171 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec55716195d4a65b48d163a1edb36f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:42.169579 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:42.181913 7523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:42.186479 7523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:42.186796 7523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:42.187767 7523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:42.189101 7523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:42.189229 7523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.189358 7523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:42.189455 7523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.222350 7523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44983
I20260619 13:07:42.222397 7658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44983 every 8 connection(s)
I20260619 13:07:42.223781 7523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:42.232649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7523
I20260619 13:07:42.232990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6866
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:07:42.293323 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.227005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46765 (local address 127.0.97.61:38469)
0619 13:07:42.269212 (+ 42207us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.269222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.269250 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:07:42.270171 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.270180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.270290 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.270413 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.274201 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.275795 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.278693 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.279385 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.279503 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.285316 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.285324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.285356 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.285404 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.289042 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.290052 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.290058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.290062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.290207 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.290734 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.290739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.290742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.291107 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.291240 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.292690 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.292921 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42029,"thread_start_us":45604,"threads_started":1}
I20260619 13:07:42.294435 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.226900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34501 (local address 127.0.97.62:37865)
0619 13:07:42.274296 (+ 47396us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.274305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.274796 (+ 491us) server_negotiation.cc:407] Connection header received
0619 13:07:42.274876 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.274884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.274989 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.277614 (+ 2625us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.277793 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.278932 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.281012 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.281779 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.281909 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.286190 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.286198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.286204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.286248 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.290616 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.291431 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.291438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.291441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.291597 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.292467 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.292472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.292475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.292814 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.292947 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.293829 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.294046 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47208,"thread_start_us":48646,"threads_started":1}
I20260619 13:07:42.295285 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.234789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.97.3:41925)
0619 13:07:42.264678 (+ 29889us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.264692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.264733 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:07:42.264840 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.264852 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.265170 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.265370 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.269373 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.271515 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.276899 (+ 5384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.277911 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.278037 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.286685 (+ 8648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.286703 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.286723 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.286764 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.291181 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.292069 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.292080 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.292085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.292227 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.292654 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.292658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.292661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.293064 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.293356 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.294831 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.295027 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29640,"thread_start_us":31122,"threads_started":1}
I20260619 13:07:42.295451 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.227626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.97.2:40419)
0619 13:07:42.275582 (+ 47956us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.275603 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.275706 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.277392 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.277397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.278001 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.278406 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.278419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.279851 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.279856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.281497 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.281511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.282796 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.282802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.283779 (+ 977us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.284384 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.284407 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.287115 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.291531 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.291535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.291539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.292495 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.292925 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.292929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.292932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.293072 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.293786 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.293790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.293908 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.294032 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.294269 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47780,"mutex_wait_us":222,"thread_start_us":41742,"threads_started":1}
I20260619 13:07:42.296373 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.227770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:34501)
0619 13:07:42.266191 (+ 38421us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.266215 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.269185 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.275859 (+ 6674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.275865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.276356 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.276723 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.276740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.279096 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.279103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.280775 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.280792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.282166 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.282174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.282804 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.283444 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.283472 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.286018 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.290793 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.290800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.290804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.291298 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.291744 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.291748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.291753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.292331 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.293019 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.293024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.293190 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.293366 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.295581 (+ 2215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38262,"mutex_wait_us":1673,"spinlock_wait_cycles":22912,"thread_start_us":47492,"threads_started":1}
I20260619 13:07:42.296463 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.226892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40419 (local address 127.0.97.60:44491)
0619 13:07:42.277002 (+ 50110us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.277010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.277027 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:42.277078 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.277083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.277166 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.277363 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.278602 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.279695 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.281682 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.282344 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.282446 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.287686 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.287694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.287698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.287982 (+ 284us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.291408 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.292647 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.292654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.292658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.292795 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.293213 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.293217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.293220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.293554 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.293733 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.295841 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.296067 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49962,"thread_start_us":49775,"threads_started":1}
I20260619 13:07:42.296599 6989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:42.296720 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.227626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.97.2:46765)
0619 13:07:42.264464 (+ 36838us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.264530 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.269930 (+ 5400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.270999 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.271016 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.272382 (+ 1366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.273914 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.273948 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.276064 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.276077 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.278521 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.278534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.279813 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.279821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.280207 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.281043 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.281085 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.285048 (+ 3963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.289211 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.289222 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.289240 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.289915 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.290360 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.290366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.290369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.290590 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.291338 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.291348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.291814 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.293592 (+ 1778us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.293997 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33975,"mutex_wait_us":54,"thread_start_us":34054,"threads_started":1}
I20260619 13:07:42.297552 7659 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44491
I20260619 13:07:42.297907 7659 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.297909 7661 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38469
I20260619 13:07:42.298061 7661 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.298159 7660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:42.298337 7660 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.298986 7660 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:42.299167 7659 heartbeater.cc:511] Master 127.0.97.60:44491 requested a full tablet report, sending...
I20260619 13:07:42.299017 7661 heartbeater.cc:511] Master 127.0.97.61:38469 requested a full tablet report, sending...
I20260619 13:07:42.300952 7331 ts_manager.cc:194] Registered new tserver with Master: a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:42.301159 6538 ts_manager.cc:194] Re-registered known tserver with Master: a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:42.302634 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34501
I20260619 13:07:42.303393 7137 ts_manager.cc:194] Registered new tserver with Master: a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:42.382222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6866
I20260619 13:07:42.398479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41925
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37865
--webserver_port=32943
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:07:42.654107 7512 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
W20260619 13:07:42.737097 7672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:42.737464 7672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:42.737644 7672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:42.745102 7672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:42.745218 7672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:42.745337 7672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:42.755800 7672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32943
--tserver_master_addrs=127.0.97.62:37865,127.0.97.61:38469,127.0.97.60:44491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:42.757324 7672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:42.758788 7672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:42.772868 7680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:42.772833 7678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:42.772914 7677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:42.773437 7672 server_base.cc:1034] running on GCE node
I20260619 13:07:42.774228 7672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:42.775233 7672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:42.776479 7672 hybrid_clock.cc:648] HybridClock initialized: now 1781874462776410 us; error 46 us; skew 500 ppm
I20260619 13:07:42.776865 7672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:42.780661 7672 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:07:42.782215 7672 webserver.cc:460] Webserver started at http://127.0.97.3:32943/ using document root <none> and password file <none>
I20260619 13:07:42.782953 7672 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:42.783118 7672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:42.788743 7672 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:07:42.791694 7687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.793485 7672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:07:42.793735 7672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "fb6246d182494890ad91fb0576ec0233"
format_stamp: "Formatted at 2026-06-19 13:07:39 on dist-test-slave-q11v"
server_key: "42d8731adc0d6c0944b6beaf609dd581"
server_key_iv: "2b84826b3593f219346e12812275229a"
server_key_version: "encryptionkey@0"
I20260619 13:07:42.794796 7672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:42.824054 7672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:42.828233 7672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:42.828558 7672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:42.829567 7672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:42.831044 7672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:42.831162 7672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.831305 7672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:42.831393 7672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:42.865257 7672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41925
I20260619 13:07:42.865345 7799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41925 every 8 connection(s)
I20260619 13:07:42.866755 7672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:42.870921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7672
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:07:42.907945 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.869549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33157 (local address 127.0.97.60:44491)
0619 13:07:42.870008 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.870014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.883337 (+ 13323us) server_negotiation.cc:407] Connection header received
0619 13:07:42.887567 (+ 4230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.887578 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.887696 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.887838 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.890559 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.891732 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.894776 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.895421 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.895528 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.900761 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.900768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.900773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.900811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.904273 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.905206 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.905211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.905215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.905354 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.905920 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.905925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.905929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.906263 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.906382 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.907351 (+ 969us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.907579 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":380,"threads_started":1}
I20260619 13:07:42.910459 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.870336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.97.3:33157)
0619 13:07:42.883121 (+ 12785us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.883193 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.887136 (+ 3943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.888278 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.888295 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.889077 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.890310 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.890335 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.891955 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.891960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.894626 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.894635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.895881 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.895888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.896314 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.897206 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.897234 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.900512 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.904471 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.904482 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.904502 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.905070 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.905503 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.905508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.905512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.905807 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.906480 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.906487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.906903 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.908369 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.908922 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12423,"mutex_wait_us":489,"spinlock_wait_cycles":66176,"thread_start_us":12151,"threads_started":1}
I20260619 13:07:42.912348 7800 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44491
Jun 19 13:07:42 dist-test-slave-q11v krb5kdc[6493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:42.911154 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.874141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48099 (local address 127.0.97.61:38469)
0619 13:07:42.886680 (+ 12539us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.886684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.886701 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:42.887720 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.887725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.887817 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.887901 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.890521 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.891855 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.894874 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.895697 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.895810 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.902144 (+ 6334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.902154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.902161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.902213 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.906685 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.907791 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.907800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.907807 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.907975 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.908618 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.908626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.908632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.909507 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.909704 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.910350 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.910640 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11806,"thread_start_us":11682,"threads_started":1}
I20260619 13:07:42.912714 7800 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.913678 7800 heartbeater.cc:511] Master 127.0.97.60:44491 requested a full tablet report, sending...
I20260619 13:07:42.913872 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.873548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.97.3:48099)
0619 13:07:42.886506 (+ 12958us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.886521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.887576 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.888281 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.888296 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.889335 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.890316 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.890337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.892086 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.892095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.894683 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.894702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.896171 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.896180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.896486 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.897498 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.897538 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.901893 (+ 4355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.906909 (+ 5016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.906917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.906923 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.907620 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.908241 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.908249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.908255 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.908466 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.909794 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.909802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.909992 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.912846 (+ 2854us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.913221 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12857,"mutex_wait_us":301,"spinlock_wait_cycles":2816,"thread_start_us":12645,"threads_started":1}
I20260619 13:07:42.915715 7331 ts_manager.cc:194] Registered new tserver with Master: fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:42.920184 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.869858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56819 (local address 127.0.97.62:37865)
0619 13:07:42.882411 (+ 12553us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.882420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.886357 (+ 3937us) server_negotiation.cc:407] Connection header received
0619 13:07:42.889254 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.889260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.889334 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.889443 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.890773 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.891810 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.894775 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.895437 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.895528 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.913924 (+ 18396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.913951 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.913956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.913997 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.917562 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.918333 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.918339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.918342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.918464 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.918823 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.918827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.918831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.919152 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.919299 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.919773 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.919952 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12385,"thread_start_us":15771,"threads_started":1}
I20260619 13:07:42.920591 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.870336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:56819)
0619 13:07:42.886223 (+ 15887us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.886254 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.889097 (+ 2843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.889634 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.889638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.890314 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:42.890616 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.890630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.891974 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.891978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.894607 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.894625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.895881 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.895888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.896312 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.910044 (+ 13732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:42.910070 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:42.913625 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:42.917773 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.917778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.917782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.918209 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.918570 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:42.918576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:42.918580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:42.918712 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:42.919296 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:42.919304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:42.919489 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.919681 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.920032 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15695,"mutex_wait_us":438,"spinlock_wait_cycles":2688,"thread_start_us":15356,"threads_started":1}
I20260619 13:07:42.921586 7801 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37865
I20260619 13:07:42.921741 7801 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.922209 7801 heartbeater.cc:511] Master 127.0.97.62:37865 requested a full tablet report, sending...
I20260619 13:07:42.923283 6538 ts_manager.cc:194] Re-registered known tserver with Master: fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:42.924559 6538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56819
I20260619 13:07:42.931408 7802 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38469
I20260619 13:07:42.931609 7802 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:42.932150 7802 heartbeater.cc:511] Master 127.0.97.61:38469 requested a full tablet report, sending...
I20260619 13:07:42.933684 7137 ts_manager.cc:194] Registered new tserver with Master: fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:42.938617 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.890497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.0.97.62:37865)
0619 13:07:42.920481 (+ 29984us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.920487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.920503 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:07:42.920553 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.920556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.920606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.920718 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.923125 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.924837 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.926956 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.927737 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.927878 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.931267 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.931274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.931279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.931319 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.935113 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.936462 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.936467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.936474 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.936617 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.937027 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.937033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.937038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.937371 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.937632 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.938008 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.938250 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29897,"thread_start_us":43699,"threads_started":1}
I20260619 13:07:42.949059 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.907944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.97.61:38469)
0619 13:07:42.912792 (+ 4848us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.912801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.930323 (+ 17522us) server_negotiation.cc:407] Connection header received
0619 13:07:42.930887 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.930892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.931126 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.931199 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.934473 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.935361 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.937276 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.937871 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.937981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.942114 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.942124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.942131 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.942173 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.945310 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.946130 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.946136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.946139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.946301 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.946735 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.946741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.946745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.947736 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.948025 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.948410 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.948667 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4694,"thread_start_us":22579,"threads_started":1}
I20260619 13:07:42.951242 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.917127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.0.97.60:44491)
0619 13:07:42.917305 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.917311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.934401 (+ 17090us) server_negotiation.cc:407] Connection header received
0619 13:07:42.934511 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.934519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.934608 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.934709 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:42.936908 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.937958 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.940610 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.941209 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.941341 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.944822 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:42.944829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:42.944834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:42.944878 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:42.947810 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.948682 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.948691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.948696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.948868 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:42.949367 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:42.949372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:42.949376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:42.949822 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:42.950048 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.950679 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.950928 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:07:42.944839 6538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:42.954727 6538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:42.990568 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.973864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32951 (local address 127.0.0.1:46884)
0619 13:07:42.981774 (+ 7910us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.981801 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.981998 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.983310 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.983314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.983339 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:42.983635 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.983643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.984959 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.984965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.987609 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.987623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.989717 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.989723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.989936 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.989993 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.990106 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.990239 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7739,"thread_start_us":14681,"threads_started":1}
I20260619 13:07:42.991034 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.974154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.0.97.1:32951)
0619 13:07:42.981192 (+ 7038us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.981206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.981909 (+ 703us) server_negotiation.cc:407] Connection header received
0619 13:07:42.982184 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.982196 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.982565 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.983294 (+ 729us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:42.983815 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.984822 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.987772 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.989271 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.989348 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.989510 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.990177 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.990374 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6713,"thread_start_us":6724,"threads_started":1}
I20260619 13:07:42.996989 7446 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 (DEFAULT_TABLE table=test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:42.998039 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.974721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41925 (local address 127.0.0.1:41656)
0619 13:07:42.988809 (+ 14088us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.988826 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.988943 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.989622 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.989629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.989660 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:42.990023 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.990035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.993233 (+ 3198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.993239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.995032 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.995045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.996769 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.996775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.996938 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.996978 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 13:07:42.997090 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 13:07:42.997220 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13959,"thread_start_us":6710,"threads_started":1}
I20260619 13:07:42.998044 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.975418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.97.3:41925)
0619 13:07:42.982660 (+ 7242us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.982666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.988964 (+ 6298us) server_negotiation.cc:407] Connection header received
0619 13:07:42.989087 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.989095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.989352 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.989635 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:42.991550 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.993072 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.995409 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.996576 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.996741 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.996941 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:42.997142 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:07:42.997294 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7086,"thread_start_us":6850,"threads_started":1}
I20260619 13:07:42.998526 7446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbe9319ed764cb1afb696dc8f38abb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:43.002048 7735 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 (DEFAULT_TABLE table=test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:43.003522 7735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbe9319ed764cb1afb696dc8f38abb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:43.006492 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.974642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44983 (local address 127.0.0.1:35394)
0619 13:07:42.986774 (+ 12132us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:42.986793 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:42.986908 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:42.987820 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:42.987826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:42.987852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:42.988284 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.988297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.990219 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.990225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:42.992451 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:42.992468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.994125 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.994133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.005789 (+ 11656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.005822 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:07:43.005959 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 13:07:43.006130 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12014,"thread_start_us":11888,"threads_started":1}
I20260619 13:07:43.006485 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:42.974918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.0.97.2:44983)
0619 13:07:42.986355 (+ 11437us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:42.986365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:42.987030 (+ 665us) server_negotiation.cc:407] Connection header received
0619 13:07:42.987135 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:42.987145 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:42.987414 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:42.987616 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:42.988496 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.990015 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.992631 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:42.993884 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:42.994037 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:42.994239 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.006032 (+ 11793us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.006226 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11204,"thread_start_us":11117,"threads_started":1}
I20260619 13:07:43.012925 7594 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 (DEFAULT_TABLE table=test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:43.014775 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbe9319ed764cb1afb696dc8f38abb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:43.016642 7832 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Bootstrap starting.
I20260619 13:07:43.020344 7833 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Bootstrap starting.
I20260619 13:07:43.020395 7832 tablet_bootstrap.cc:654] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:43.022147 7832 log.cc:824] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:43.024263 7833 tablet_bootstrap.cc:654] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:43.025549 7832 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: No bootstrap required, opened a new log
I20260619 13:07:43.025520 7833 log.cc:824] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:43.025864 7832 ts_tablet_manager.cc:1397] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260619 13:07:43.029143 7833 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: No bootstrap required, opened a new log
I20260619 13:07:43.029412 7833 ts_tablet_manager.cc:1397] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:07:43.029656 7836 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Bootstrap starting.
I20260619 13:07:43.032485 7836 tablet_bootstrap.cc:654] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:43.033689 7836 log.cc:824] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:43.036875 7832 raft_consensus.cc:348] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.037372 7832 raft_consensus.cc:374] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:43.037473 7832 raft_consensus.cc:729] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96950338f6c44e0282ba11e5ea2e0289, State: Initialized, Role: FOLLOWER
I20260619 13:07:43.038160 7832 consensus_queue.cc:260] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.038928 7836 tablet_bootstrap.cc:492] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: No bootstrap required, opened a new log
I20260619 13:07:43.039220 7836 ts_tablet_manager.cc:1397] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 13:07:43.039933 7832 ts_tablet_manager.cc:1428] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260619 13:07:43.039934 7833 raft_consensus.cc:348] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.040361 7833 raft_consensus.cc:374] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:43.040457 7833 raft_consensus.cc:729] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6246d182494890ad91fb0576ec0233, State: Initialized, Role: FOLLOWER
I20260619 13:07:43.041180 7833 consensus_queue.cc:260] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.046658 7801 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
I20260619 13:07:43.049147 7833 ts_tablet_manager.cc:1428] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260619 13:07:43.052243 7836 raft_consensus.cc:348] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.052655 7836 raft_consensus.cc:374] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:43.052763 7836 raft_consensus.cc:729] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77f5f40817949d4a40cd63d82b75135, State: Initialized, Role: FOLLOWER
I20260619 13:07:43.053345 7836 consensus_queue.cc:260] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.054503 7660 heartbeater.cc:503] Master 127.0.97.62:37865 was elected leader, sending a full tablet report...
I20260619 13:07:43.055238 7836 ts_tablet_manager.cc:1428] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260619 13:07:43.103808 7514 tablet.cc:2369] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:43.125943 7803 tablet.cc:2369] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:43.227347 7662 tablet.cc:2369] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:43.283264 7839 raft_consensus.cc:482] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:43.283545 7839 raft_consensus.cc:504] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.285343 7839 leader_election.cc:288] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951), a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:43.291095 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.285897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40087 (local address 127.0.97.2:44983)
0619 13:07:43.286178 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.286182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.286197 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:07:43.286255 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.286260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.286312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.286419 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:43.286812 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.287512 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.289179 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.290049 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.290216 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.290317 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.290677 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.290821 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260619 13:07:43.291116 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.285939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55739 (local address 127.0.97.1:32951)
0619 13:07:43.286211 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.286219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.286240 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:07:43.286314 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.286317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.286382 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.286490 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:43.286849 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.287621 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.289214 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.290159 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.290314 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.290422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.290705 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.290840 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 13:07:43.291167 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.285633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44983 (local address 127.0.97.3:40087)
0619 13:07:43.285825 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:43.285848 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:43.285973 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:43.286435 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:43.286439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:43.286461 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:43.286657 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.286666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.287647 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.287654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.289006 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.289018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.290245 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.290251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.290511 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.290537 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 13:07:43.290651 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 13:07:43.290880 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":52}
I20260619 13:07:43.291428 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.285643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32951 (local address 127.0.97.3:55739)
0619 13:07:43.286055 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:43.286073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:43.286174 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:43.286505 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:43.286509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:43.286532 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:43.286702 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.286710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.287754 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.287760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.289058 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.289069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.290341 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.290351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.290555 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.290577 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:07:43.290681 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:07:43.291139 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":328}
I20260619 13:07:43.291590 7838 raft_consensus.cc:482] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:43.291893 7838 raft_consensus.cc:504] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.292589 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "fb6246d182494890ad91fb0576ec0233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77f5f40817949d4a40cd63d82b75135" is_pre_election: true
I20260619 13:07:43.293138 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "fb6246d182494890ad91fb0576ec0233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96950338f6c44e0282ba11e5ea2e0289" is_pre_election: true
I20260619 13:07:43.293565 7614 raft_consensus.cc:2454] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6246d182494890ad91fb0576ec0233 in term 0.
I20260619 13:07:43.293757 7466 raft_consensus.cc:2454] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6246d182494890ad91fb0576ec0233 in term 0.
I20260619 13:07:43.294329 7688 leader_election.cc:302] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a77f5f40817949d4a40cd63d82b75135, fb6246d182494890ad91fb0576ec0233; no voters:
I20260619 13:07:43.294680 7838 leader_election.cc:288] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925), a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:43.295006 7839 raft_consensus.cc:2781] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:43.295653 7839 raft_consensus.cc:482] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:43.295782 7839 raft_consensus.cc:3037] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:43.298789 7839 raft_consensus.cc:504] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.299943 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "fb6246d182494890ad91fb0576ec0233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96950338f6c44e0282ba11e5ea2e0289"
I20260619 13:07:43.300177 7466 raft_consensus.cc:3037] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:43.300168 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "fb6246d182494890ad91fb0576ec0233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77f5f40817949d4a40cd63d82b75135"
I20260619 13:07:43.300367 7614 raft_consensus.cc:3037] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:43.304034 7466 raft_consensus.cc:2454] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6246d182494890ad91fb0576ec0233 in term 1.
I20260619 13:07:43.304178 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.294948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44983 (local address 127.0.97.1:58151)
0619 13:07:43.295927 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:43.295958 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:43.296182 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:43.296582 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:43.296591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:43.296634 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:43.297226 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.297246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.298738 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.298744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.301079 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.301096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.303317 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.303323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.303531 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.303569 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:07:43.303636 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 13:07:43.303770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":425,"threads_started":1}
I20260619 13:07:43.304150 7614 raft_consensus.cc:2454] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6246d182494890ad91fb0576ec0233 in term 1.
I20260619 13:07:43.304590 7688 leader_election.cc:302] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a77f5f40817949d4a40cd63d82b75135, fb6246d182494890ad91fb0576ec0233; no voters:
I20260619 13:07:43.304986 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.295550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58151 (local address 127.0.97.2:44983)
0619 13:07:43.295723 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.295730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.296068 (+ 338us) server_negotiation.cc:407] Connection header received
0619 13:07:43.296346 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.296351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.296428 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.296532 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:43.297433 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.298560 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.301295 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.303059 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.304402 (+ 1343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.304514 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.304563 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.304701 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:07:43.305238 7843 raft_consensus.cc:2781] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:43.305467 7839 leader_election.cc:288] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [CANDIDATE]: Term 1 election: Requested vote from peers 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951), a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983)
I20260619 13:07:43.305639 7613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "96950338f6c44e0282ba11e5ea2e0289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77f5f40817949d4a40cd63d82b75135" is_pre_election: true
I20260619 13:07:43.306214 7613 raft_consensus.cc:2379] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96950338f6c44e0282ba11e5ea2e0289 in current term 1: Already voted for candidate fb6246d182494890ad91fb0576ec0233 in this term.
I20260619 13:07:43.306285 7843 raft_consensus.cc:686] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 1 LEADER]: Becoming Leader. State: Replica: fb6246d182494890ad91fb0576ec0233, State: Running, Role: LEADER
I20260619 13:07:43.306944 7843 consensus_queue.cc:237] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:43.307576 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.295201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41925 (local address 127.0.97.1:44015)
0619 13:07:43.298107 (+ 2906us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:43.298124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:43.298290 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:43.298630 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:43.298635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:43.298667 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:43.299154 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.299170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.300825 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.300834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.303339 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:43.303356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.306804 (+ 3448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.306811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:43.307027 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.307060 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:07:43.307158 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:07:43.307320 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2786,"thread_start_us":2636,"threads_started":1}
I20260619 13:07:43.307601 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.295244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44015 (local address 127.0.97.3:41925)
0619 13:07:43.295434 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.295439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.298227 (+ 2788us) server_negotiation.cc:407] Connection header received
0619 13:07:43.298422 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.298428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.298503 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.298603 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:43.299374 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.300523 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.303538 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.304903 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.305049 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.305163 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.307220 (+ 2057us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.307363 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:07:43.308538 7755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbe9319ed764cb1afb696dc8f38abb5" candidate_uuid: "96950338f6c44e0282ba11e5ea2e0289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6246d182494890ad91fb0576ec0233" is_pre_election: true
I20260619 13:07:43.309525 7401 leader_election.cc:302] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96950338f6c44e0282ba11e5ea2e0289; no voters: a77f5f40817949d4a40cd63d82b75135, fb6246d182494890ad91fb0576ec0233
I20260619 13:07:43.310377 7838 raft_consensus.cc:2726] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 13:07:43.312727 6537 catalog_manager.cc:5495] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb6246d182494890ad91fb0576ec0233 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb6246d182494890ad91fb0576ec0233" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:43.363924 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.358769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.3:41925)
0619 13:07:43.358930 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.358935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.358974 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:07:43.359162 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.359166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.359256 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.359380 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:43.359684 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.360439 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.361779 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.362263 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.362444 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.362880 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:43.363408 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:43.363529 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.363630 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.363707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:07:43.372328 7613 raft_consensus.cc:1261] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Refusing update from remote peer fb6246d182494890ad91fb0576ec0233: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:43.372381 7466 raft_consensus.cc:1261] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Refusing update from remote peer fb6246d182494890ad91fb0576ec0233: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:43.373317 7844 consensus_queue.cc:1035] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:43.373731 7843 consensus_queue.cc:1035] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:43.387868 7848 mvcc.cc:205] Tried to move back new op lower bound from 7298557801962639360 to 7298557801710325760. Current Snapshot: MvccSnapshot[applied={T|T < 7298557801962639360}]
I20260619 13:07:43.400135 7849 mvcc.cc:205] Tried to move back new op lower bound from 7298557801962639360 to 7298557801710325760. Current Snapshot: MvccSnapshot[applied={T|T < 7298557801962639360}]
I20260619 13:07:43.407157 6538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:43.407476 6538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:43.418452 6538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec55716195d4a65b48d163a1edb36f4: Sending DeleteTablet for 3 replicas of tablet 4dbe9319ed764cb1afb696dc8f38abb5
I20260619 13:07:43.419621 7446 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46884
I20260619 13:07:43.420573 7857 tablet_replica.cc:330] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: stopping tablet replica
I20260619 13:07:43.421126 7857 raft_consensus.cc:2229] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:43.421758 7857 raft_consensus.cc:2258] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:43.423089 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35394
I20260619 13:07:43.423672 7593 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35394
I20260619 13:07:43.424145 7857 ts_tablet_manager.cc:1905] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:43.426111 7735 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbe9319ed764cb1afb696dc8f38abb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:43 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41656
I20260619 13:07:43.432334 7859 tablet_replica.cc:330] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: stopping tablet replica
I20260619 13:07:43.432843 7859 raft_consensus.cc:2229] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:43.433401 7859 raft_consensus.cc:2258] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:43.433784 7857 ts_tablet_manager.cc:1918] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:43.434064 7857 log.cc:1196] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/4dbe9319ed764cb1afb696dc8f38abb5
I20260619 13:07:43.434777 7857 ts_tablet_manager.cc:1939] T 4dbe9319ed764cb1afb696dc8f38abb5 P 96950338f6c44e0282ba11e5ea2e0289: Deleting consensus metadata
I20260619 13:07:43.436911 6523 catalog_manager.cc:4841] TS 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951): tablet 4dbe9319ed764cb1afb696dc8f38abb5 (table test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]) successfully deleted
I20260619 13:07:43.439090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 6506
I20260619 13:07:43.440161 6568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:43.440239 7859 ts_tablet_manager.cc:1905] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:43.442436 7860 tablet_replica.cc:330] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: stopping tablet replica
I20260619 13:07:43.443042 7860 raft_consensus.cc:2229] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:43.443797 7860 raft_consensus.cc:2258] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:43.449272 7860 ts_tablet_manager.cc:1905] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:43.454455 7859 ts_tablet_manager.cc:1918] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:43.454713 7859 log.cc:1196] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/4dbe9319ed764cb1afb696dc8f38abb5
I20260619 13:07:43.455343 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.422309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.0.97.61:38469)
0619 13:07:43.422467 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.422473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.439979 (+ 17506us) server_negotiation.cc:407] Connection header received
0619 13:07:43.440192 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.440198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.440286 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.440379 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:43.442710 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.443585 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.445024 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.445427 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.445568 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.448309 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:43.448315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:43.448320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:43.448363 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:43.451750 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.452414 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.452420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.452424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.452564 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.452945 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.452950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.452953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.453243 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:43.453509 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.454532 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.454913 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:07:43.455394 7859 ts_tablet_manager.cc:1939] T 4dbe9319ed764cb1afb696dc8f38abb5 P a77f5f40817949d4a40cd63d82b75135: Deleting consensus metadata
I20260619 13:07:43.461293 7860 ts_tablet_manager.cc:1918] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:43.461480 7860 log.cc:1196] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/4dbe9319ed764cb1afb696dc8f38abb5
I20260619 13:07:43.462092 7860 ts_tablet_manager.cc:1939] T 4dbe9319ed764cb1afb696dc8f38abb5 P fb6246d182494890ad91fb0576ec0233: Deleting consensus metadata
I20260619 13:07:43.474877 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.438386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.0.97.60:44491)
0619 13:07:43.438605 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.438611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.457334 (+ 18723us) server_negotiation.cc:407] Connection header received
0619 13:07:43.457621 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.457627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.457717 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.457797 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:43.459718 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.461031 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.462985 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.463623 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.463759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.467169 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:43.467177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:43.467182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:43.467230 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:43.471116 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.472036 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.472042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.472046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.472206 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.472647 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.472653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.472656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.473037 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:43.473299 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.473739 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.474393 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 13:07:43.582747 6522 catalog_manager.cc:4841] TS a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983): tablet 4dbe9319ed764cb1afb696dc8f38abb5 (table test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]) successfully deleted
I20260619 13:07:43.583388 6524 catalog_manager.cc:4841] TS fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925): tablet 4dbe9319ed764cb1afb696dc8f38abb5 (table test-table [id=ec8f847ef62c4ebcb22c90a8a0287aac]) successfully deleted
W20260619 13:07:43.583413 6522 catalog_manager.cc:4808] TS a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983): delete failed for tablet 4dbe9319ed764cb1afb696dc8f38abb5 because the tablet was not found. No further retry: Not found: Tablet not found: 4dbe9319ed764cb1afb696dc8f38abb5
I20260619 13:07:43.584102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 6506
I20260619 13:07:43.641094 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.621096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.0.97.60:44491)
0619 13:07:43.621320 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.621326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.626191 (+ 4865us) server_negotiation.cc:407] Connection header received
0619 13:07:43.626444 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.626452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.626552 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.626658 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:43.628729 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.629572 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.631037 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.631730 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.631863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.635382 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:43.635392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:43.635395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:43.635429 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:43.638241 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.639083 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.639089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.639092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.639237 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.639635 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.639640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.639645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.639948 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:43.640211 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.640529 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.640732 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:07:43.641779 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:43.621419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.0.97.61:38469)
0619 13:07:43.621649 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:43.621654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:43.625459 (+ 3805us) server_negotiation.cc:407] Connection header received
0619 13:07:43.625863 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:43.625870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:43.625965 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:43.626088 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:43.628012 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.628859 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.630920 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:43.631393 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:43.631534 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:43.635148 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:43.635155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:43.635160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:43.635204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:43.638777 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.639435 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.639440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.639444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.639585 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:43.640026 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:43.640031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:43.640034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:43.640410 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:43.640629 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:43.641192 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 13:07:43.641412 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:07:44.396982 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.396300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:53711)
0619 13:07:44.396713 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.396822 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
W20260619 13:07:44.397451 7512 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.404636 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.403901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:35841)
0619 13:07:44.404365 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.404487 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
W20260619 13:07:44.405210 7801 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.427456 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.426760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:34993)
0619 13:07:44.427181 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.427285 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
W20260619 13:07:44.427845 7660 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.917290 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ec55716195d4a65b48d163a1edb36f4)
I20260619 13:07:44.917572 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:44.917748 7873 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ec55716195d4a65b48d163a1edb36f4)
I20260619 13:07:44.917979 7873 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:44.919068 7872 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60:44491)
I20260619 13:07:44.926692 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.919164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52982)
0619 13:07:44.926107 (+ 6943us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.926315 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6765,"threads_started":1}
W20260619 13:07:44.930685 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.933674 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.932866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52988)
0619 13:07:44.933088 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.933336 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 13:07:44.933926 7315 leader_election.cc:334] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.934701 7873 leader_election.cc:288] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61:38469)
I20260619 13:07:44.935016 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.919269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52974)
0619 13:07:44.934370 (+ 15101us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.934558 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14963,"thread_start_us":15241,"threads_started":1}
W20260619 13:07:44.935428 7121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.937263 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.936849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:52994)
0619 13:07:44.937036 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.937117 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 13:07:44.937726 7121 leader_election.cc:334] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.940883 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.919537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.0.97.60:44491)
0619 13:07:44.934282 (+ 14745us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:44.934288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:44.934868 (+ 580us) server_negotiation.cc:407] Connection header received
0619 13:07:44.935169 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:44.935176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:44.935248 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:44.935363 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:44.936046 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.937098 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.939051 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.939830 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.939971 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:44.940062 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:44.940353 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 13:07:44.940486 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":13942,"thread_start_us":14365,"threads_started":1}
I20260619 13:07:44.940886 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.919559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44491 (local address 127.0.0.1:54550)
0619 13:07:44.934761 (+ 15202us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.934781 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:44.934991 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:44.935396 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:44.935403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:44.935446 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:44.935748 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:44.935762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.937220 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.937228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:44.938898 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:44.938909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.939996 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.940002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:44.940172 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:44.940222 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:07:44.940326 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:07:44.940461 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15081,"mutex_wait_us":2,"thread_start_us":14465,"threads_started":1}
I20260619 13:07:44.941188 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.925559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38469 (local address 127.0.0.1:34242)
0619 13:07:44.926884 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.926908 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:44.927068 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:44.935491 (+ 8423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:44.935496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:44.935521 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:44.935768 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:44.935777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.937761 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.937766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:44.939082 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:44.939093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.940237 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.940244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:44.940447 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:44.940469 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:07:44.940571 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:07:44.940715 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":3,"thread_start_us":3117,"threads_started":1}
I20260619 13:07:44.941551 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.925968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.97.61:38469)
0619 13:07:44.935177 (+ 9209us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:44.935182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:44.935198 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:07:44.935248 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:44.935254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:44.935322 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:44.935926 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:44.936279 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.937638 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.939245 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:44.940061 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:44.940263 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:44.940376 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:44.940595 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 13:07:44.940722 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":9080,"thread_start_us":8893,"threads_started":1}
I20260619 13:07:44.941565 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed7c6ab783e4615811805187ba5c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb4fcd5fc7594f52b70c699e175048ad" is_pre_election: true
I20260619 13:07:44.941998 7341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed7c6ab783e4615811805187ba5c5f2 in term 1.
I20260619 13:07:44.942147 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb4fcd5fc7594f52b70c699e175048ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ed7c6ab783e4615811805187ba5c5f2" is_pre_election: true
I20260619 13:07:44.942474 7146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4fcd5fc7594f52b70c699e175048ad in term 1.
I20260619 13:07:44.942519 7124 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ed7c6ab783e4615811805187ba5c5f2, eb4fcd5fc7594f52b70c699e175048ad; no voters: 2ec55716195d4a65b48d163a1edb36f4
I20260619 13:07:44.942924 7872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 13:07:44.943053 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2ec55716195d4a65b48d163a1edb36f4)
I20260619 13:07:44.943027 7317 leader_election.cc:302] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ed7c6ab783e4615811805187ba5c5f2, eb4fcd5fc7594f52b70c699e175048ad; no voters: 2ec55716195d4a65b48d163a1edb36f4
I20260619 13:07:44.943158 7872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:07:44.943423 7873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 13:07:44.943516 7873 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2ec55716195d4a65b48d163a1edb36f4)
I20260619 13:07:44.943635 7873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:07:44.945331 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:44.945672 7873 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:44.945963 7872 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 election: Requested vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), eb4fcd5fc7594f52b70c699e175048ad (127.0.97.60:44491)
I20260619 13:07:44.946414 7873 leader_election.cc:288] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 election: Requested vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61:38469)
I20260619 13:07:44.946419 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed7c6ab783e4615811805187ba5c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb4fcd5fc7594f52b70c699e175048ad"
I20260619 13:07:44.946779 7341 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ed7c6ab783e4615811805187ba5c5f2 in current term 2: Already voted for candidate eb4fcd5fc7594f52b70c699e175048ad in this term.
I20260619 13:07:44.946926 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.946460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53026)
0619 13:07:44.946652 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.946721 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 13:07:44.946908 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb4fcd5fc7594f52b70c699e175048ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ed7c6ab783e4615811805187ba5c5f2"
I20260619 13:07:44.947211 7146 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb4fcd5fc7594f52b70c699e175048ad in current term 2: Already voted for candidate 1ed7c6ab783e4615811805187ba5c5f2 in this term.
I20260619 13:07:44.947808 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.946086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53010)
0619 13:07:44.946809 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.946912 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260619 13:07:44.947890 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.947535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53032)
0619 13:07:44.947679 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.947767 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:07:44.948050 7315 leader_election.cc:334] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.948212 7315 leader_election.cc:302] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4fcd5fc7594f52b70c699e175048ad; no voters: 1ed7c6ab783e4615811805187ba5c5f2, 2ec55716195d4a65b48d163a1edb36f4
I20260619 13:07:44.948490 7873 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 13:07:44.948982 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:44.948380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53048)
0619 13:07:44.948746 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:44.948832 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
W20260619 13:07:44.949173 7121 leader_election.cc:334] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:44.949340 7121 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ed7c6ab783e4615811805187ba5c5f2; no voters: 2ec55716195d4a65b48d163a1edb36f4, eb4fcd5fc7594f52b70c699e175048ad
I20260619 13:07:44.949571 7872 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 13:07:45.399035 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:45.398236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:39207)
0619 13:07:45.398772 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:45.398860 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":324,"threads_started":1}
I20260619 13:07:45.406915 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:45.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:52429)
0619 13:07:45.406516 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:45.406663 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":446,"threads_started":1}
I20260619 13:07:45.429265 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:45.428582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:53085)
0619 13:07:45.429003 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:45.429085 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
I20260619 13:07:46.401084 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.400131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:47873)
0619 13:07:46.400717 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.400847 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
I20260619 13:07:46.408427 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.407661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:48907)
0619 13:07:46.408152 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.408251 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":279,"threads_started":1}
I20260619 13:07:46.430948 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.430153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:48489)
0619 13:07:46.430641 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.430744 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260619 13:07:46.961414 7888 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:46.961730 7888 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:46.962353 7888 leader_election.cc:288] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61:38469)
I20260619 13:07:46.962947 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb4fcd5fc7594f52b70c699e175048ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ed7c6ab783e4615811805187ba5c5f2" is_pre_election: true
I20260619 13:07:46.963214 7146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4fcd5fc7594f52b70c699e175048ad in term 2.
I20260619 13:07:46.963543 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.962611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53322)
0619 13:07:46.963200 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.963301 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":248,"threads_started":1}
I20260619 13:07:46.963701 7317 leader_election.cc:302] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed7c6ab783e4615811805187ba5c5f2, eb4fcd5fc7594f52b70c699e175048ad; no voters:
I20260619 13:07:46.964038 7888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 13:07:46.964185 7888 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:46.964337 7888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:07:46.964957 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.964640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53330)
0619 13:07:46.964785 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.964852 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:07:46.965196 7315 leader_election.cc:334] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:46.966780 7888 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:46.967365 7888 leader_election.cc:288] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 election: Requested vote from peers 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865), 1ed7c6ab783e4615811805187ba5c5f2 (127.0.97.61:38469)
I20260619 13:07:46.967850 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.967435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53346)
0619 13:07:46.967588 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.967682 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:07:46.967902 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb4fcd5fc7594f52b70c699e175048ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ed7c6ab783e4615811805187ba5c5f2"
I20260619 13:07:46.968166 7146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:07:46.968812 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:46.968458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53360)
0619 13:07:46.968601 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:46.968691 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:07:46.968984 7315 leader_election.cc:334] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
I20260619 13:07:46.970583 7146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4fcd5fc7594f52b70c699e175048ad in term 3.
I20260619 13:07:46.971065 7317 leader_election.cc:302] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ed7c6ab783e4615811805187ba5c5f2, eb4fcd5fc7594f52b70c699e175048ad; no voters: 2ec55716195d4a65b48d163a1edb36f4
I20260619 13:07:46.971334 7888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:07:46.971766 7888 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [term 3 LEADER]: Becoming Leader. State: Replica: eb4fcd5fc7594f52b70c699e175048ad, State: Running, Role: LEADER
I20260619 13:07:46.972211 7888 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } }
I20260619 13:07:46.973788 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb4fcd5fc7594f52b70c699e175048ad. Latest consensus state: current_term: 3 leader_uuid: "eb4fcd5fc7594f52b70c699e175048ad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:46.974162 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:46.974835 7893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:46.978585 7893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:46.979439 7893 catalog_manager.cc:1259] Loaded cluster ID: 720167b46e694fa9abdcff7be8bebb3a
I20260619 13:07:46.979557 7893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:46.981333 7893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:46.982194 7893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Loaded TSK: 0
I20260619 13:07:46.983080 7893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:07:47.075021 7331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:47.082510 7331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:47.086761 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.086286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53396)
0619 13:07:47.086530 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.086614 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:07:47.086789 7146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [term 3 FOLLOWER]: Refusing update from remote peer eb4fcd5fc7594f52b70c699e175048ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 13:07:47.087543 7891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 13:07:47.087809 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.087461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53400)
0619 13:07:47.087604 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.087676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 13:07:47.087970 7315 consensus_peers.cc:487] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad -> Peer 2ec55716195d4a65b48d163a1edb36f4 (127.0.97.62:37865): Couldn't send request to peer 2ec55716195d4a65b48d163a1edb36f4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:07:47.089881 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb4fcd5fc7594f52b70c699e175048ad. Latest consensus state: current_term: 3 leader_uuid: "eb4fcd5fc7594f52b70c699e175048ad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:47.090152 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:47.091886 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "eb4fcd5fc7594f52b70c699e175048ad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:47.092233 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:47.093209 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "eb4fcd5fc7594f52b70c699e175048ad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ec55716195d4a65b48d163a1edb36f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37865 } } peers { permanent_uuid: "1ed7c6ab783e4615811805187ba5c5f2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38469 } attrs { promote: false } } peers { permanent_uuid: "eb4fcd5fc7594f52b70c699e175048ad" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44491 } attrs { promote: false } } }
I20260619 13:07:47.093482 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed7c6ab783e4615811805187ba5c5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:07:47.120150 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.101119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32951 (local address 127.0.0.1:46888)
0619 13:07:47.101288 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.101309 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:47.101426 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:47.109157 (+ 7731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:47.109165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:47.109206 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:47.109647 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.109664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.113321 (+ 3657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.113331 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.116233 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.116257 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.119276 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.119286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.119499 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.119548 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 13:07:47.119638 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:07:47.119786 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:07:47.120981 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.101414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.0.97.1:32951)
0619 13:07:47.108081 (+ 6667us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.108090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.108118 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:07:47.108215 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.108223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.108326 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.109853 (+ 1527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:47.110079 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.111410 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.117736 (+ 6326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.119071 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.120379 (+ 1308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.120488 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.120545 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.120707 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6497,"thread_start_us":531,"threads_started":1}
I20260619 13:07:47.121783 7446 tablet_service.cc:1467] Processing CreateTablet for tablet 8777f968de124d0f860c05d8e6bb1241 (DEFAULT_TABLE table=test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:47.122262 7446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8777f968de124d0f860c05d8e6bb1241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:47.125209 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.101915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.97.3:41925)
0619 13:07:47.114030 (+ 12115us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.114038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.118279 (+ 4241us) server_negotiation.cc:407] Connection header received
0619 13:07:47.118474 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.118481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.118584 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.118689 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:47.119543 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.120580 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.122758 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.124014 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.124280 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.124375 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.124775 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.124910 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11987,"thread_start_us":15538,"threads_started":1}
I20260619 13:07:47.125200 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.101814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41925 (local address 127.0.0.1:41676)
0619 13:07:47.118170 (+ 16356us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.118187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:47.118334 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:47.118894 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:47.118902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:47.118931 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:47.119325 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.119334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.120733 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.120740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.122616 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.122626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.124490 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.124495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.124618 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.124640 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:07:47.124739 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:07:47.124859 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16232,"thread_start_us":16761,"threads_started":1}
I20260619 13:07:47.126573 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.102071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44983 (local address 127.0.0.1:35398)
0619 13:07:47.118842 (+ 16771us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.118863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:47.119027 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:47.119452 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:47.119459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:47.119486 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:47.119831 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.119842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.122113 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.122118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.124465 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.124482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.126070 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.126075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.126192 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.126214 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:07:47.126285 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 13:07:47.126365 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16631,"thread_start_us":15846,"threads_started":1}
I20260619 13:07:47.126839 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.102508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.2:44983)
0619 13:07:47.117701 (+ 15193us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.117710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.118932 (+ 1222us) server_negotiation.cc:407] Connection header received
0619 13:07:47.119222 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.119230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.119311 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.119417 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:47.120212 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.121963 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.124661 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.125890 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.126024 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.126147 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.126342 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.126489 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15000,"thread_start_us":14905,"threads_started":1}
I20260619 13:07:47.127556 7594 tablet_service.cc:1467] Processing CreateTablet for tablet 8777f968de124d0f860c05d8e6bb1241 (DEFAULT_TABLE table=test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:47.128014 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8777f968de124d0f860c05d8e6bb1241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:47.129237 7735 tablet_service.cc:1467] Processing CreateTablet for tablet 8777f968de124d0f860c05d8e6bb1241 (DEFAULT_TABLE table=test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:47.129791 7735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8777f968de124d0f860c05d8e6bb1241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:47.133677 7907 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Bootstrap starting.
I20260619 13:07:47.134110 7906 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Bootstrap starting.
I20260619 13:07:47.135167 7907 tablet_bootstrap.cc:654] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:47.135812 7906 tablet_bootstrap.cc:654] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:47.143478 7907 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: No bootstrap required, opened a new log
I20260619 13:07:47.143636 7907 ts_tablet_manager.cc:1397] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260619 13:07:47.144395 7907 raft_consensus.cc:348] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.144640 7907 raft_consensus.cc:374] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:47.144726 7907 raft_consensus.cc:729] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77f5f40817949d4a40cd63d82b75135, State: Initialized, Role: FOLLOWER
I20260619 13:07:47.144994 7907 consensus_queue.cc:260] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.147174 7906 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: No bootstrap required, opened a new log
I20260619 13:07:47.147432 7906 ts_tablet_manager.cc:1397] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.011s
I20260619 13:07:47.148708 7906 raft_consensus.cc:348] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.148998 7906 raft_consensus.cc:374] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:47.149112 7906 raft_consensus.cc:729] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96950338f6c44e0282ba11e5ea2e0289, State: Initialized, Role: FOLLOWER
I20260619 13:07:47.149399 7906 consensus_queue.cc:260] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.150275 7907 ts_tablet_manager.cc:1428] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260619 13:07:47.156903 7910 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Bootstrap starting.
I20260619 13:07:47.160624 7910 tablet_bootstrap.cc:654] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:47.161770 7659 heartbeater.cc:503] Master 127.0.97.60:44491 was elected leader, sending a full tablet report...
I20260619 13:07:47.165246 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.151451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:55041)
0619 13:07:47.164791 (+ 13340us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.164921 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12942,"thread_start_us":13100,"threads_started":1}
I20260619 13:07:47.166258 7906 ts_tablet_manager.cc:1428] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260619 13:07:47.166880 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.166040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:55715)
0619 13:07:47.166641 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.166725 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"threads_started":1}
I20260619 13:07:47.168236 7511 heartbeater.cc:503] Master 127.0.97.60:44491 was elected leader, sending a full tablet report...
I20260619 13:07:47.171509 7910 tablet_bootstrap.cc:492] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: No bootstrap required, opened a new log
I20260619 13:07:47.171723 7910 ts_tablet_manager.cc:1397] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20260619 13:07:47.172547 7910 raft_consensus.cc:348] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.172873 7910 raft_consensus.cc:374] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:47.173005 7910 raft_consensus.cc:729] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6246d182494890ad91fb0576ec0233, State: Initialized, Role: FOLLOWER
I20260619 13:07:47.173301 7910 consensus_queue.cc:260] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.178169 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.166721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:50941)
0619 13:07:47.177773 (+ 11052us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.177890 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10879,"thread_start_us":10666,"threads_started":1}
I20260619 13:07:47.178350 7910 ts_tablet_manager.cc:1428] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 13:07:47.181325 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.179099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:39665)
0619 13:07:47.179601 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.181081 (+ 1480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":675,"threads_started":1}
I20260619 13:07:47.181474 7800 heartbeater.cc:503] Master 127.0.97.60:44491 was elected leader, sending a full tablet report...
I20260619 13:07:47.181017 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.180160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:44967)
0619 13:07:47.180785 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.180868 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 13:07:47.182337 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.181946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:43379)
0619 13:07:47.182126 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.182211 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:07:47.581811 7911 raft_consensus.cc:482] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:47.582055 7911 raft_consensus.cc:504] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.583658 7911 leader_election.cc:288] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951), fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:47.584019 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.583554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53416)
0619 13:07:47.583731 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.583847 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 13:07:47.585263 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.584894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53422)
0619 13:07:47.585074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.585140 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:07:47.589980 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.584092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45531 (local address 127.0.97.1:32951)
0619 13:07:47.584287 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.584295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.584444 (+ 149us) server_negotiation.cc:407] Connection header received
0619 13:07:47.584733 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.584740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.584814 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.584936 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:47.585589 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.586394 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.588283 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.589041 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.589187 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.589293 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.589578 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.589732 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:07:47.590225 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.584217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57177 (local address 127.0.97.3:41925)
0619 13:07:47.584411 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.584435 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.584602 (+ 167us) server_negotiation.cc:407] Connection header received
0619 13:07:47.585145 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.585152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.585230 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.585329 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:47.585928 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.586696 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.588488 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.589234 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.589386 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.589486 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.589852 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.589977 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:07:47.590274 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.583982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41925 (local address 127.0.97.2:57177)
0619 13:07:47.584470 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.584487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:47.584647 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:47.585380 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:47.585384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:47.585434 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:47.585767 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.585776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.586839 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.586847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.588333 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.588345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.589462 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.589468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.589701 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.589724 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:07:47.589828 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:07:47.589963 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":24}
I20260619 13:07:47.590057 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.583846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32951 (local address 127.0.97.2:45531)
0619 13:07:47.584313 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.584331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:47.584576 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:47.584961 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:47.584966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:47.584991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:47.585362 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.585375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.586515 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.586521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.588143 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:47.588154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.589243 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.589248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:47.589420 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.589447 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:07:47.589555 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:07:47.589718 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":45}
I20260619 13:07:47.590826 7755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8777f968de124d0f860c05d8e6bb1241" candidate_uuid: "a77f5f40817949d4a40cd63d82b75135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6246d182494890ad91fb0576ec0233" is_pre_election: true
I20260619 13:07:47.591090 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8777f968de124d0f860c05d8e6bb1241" candidate_uuid: "a77f5f40817949d4a40cd63d82b75135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96950338f6c44e0282ba11e5ea2e0289" is_pre_election: true
I20260619 13:07:47.591204 7755 raft_consensus.cc:2454] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77f5f40817949d4a40cd63d82b75135 in term 0.
I20260619 13:07:47.591336 7466 raft_consensus.cc:2454] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77f5f40817949d4a40cd63d82b75135 in term 0.
I20260619 13:07:47.591710 7547 leader_election.cc:302] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a77f5f40817949d4a40cd63d82b75135, fb6246d182494890ad91fb0576ec0233; no voters:
I20260619 13:07:47.592113 7911 raft_consensus.cc:2781] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:47.592250 7911 raft_consensus.cc:482] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:47.592363 7911 raft_consensus.cc:3037] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:47.594587 7911 raft_consensus.cc:504] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.595243 7911 leader_election.cc:288] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [CANDIDATE]: Term 1 election: Requested vote from peers 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951), fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925)
I20260619 13:07:47.595629 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8777f968de124d0f860c05d8e6bb1241" candidate_uuid: "a77f5f40817949d4a40cd63d82b75135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96950338f6c44e0282ba11e5ea2e0289"
I20260619 13:07:47.595726 7755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8777f968de124d0f860c05d8e6bb1241" candidate_uuid: "a77f5f40817949d4a40cd63d82b75135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6246d182494890ad91fb0576ec0233"
I20260619 13:07:47.595856 7466 raft_consensus.cc:3037] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:47.595973 7755 raft_consensus.cc:3037] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:47.598939 7755 raft_consensus.cc:2454] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77f5f40817949d4a40cd63d82b75135 in term 1.
I20260619 13:07:47.599097 7466 raft_consensus.cc:2454] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77f5f40817949d4a40cd63d82b75135 in term 1.
I20260619 13:07:47.599480 7546 leader_election.cc:302] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96950338f6c44e0282ba11e5ea2e0289, a77f5f40817949d4a40cd63d82b75135; no voters:
I20260619 13:07:47.599767 7911 raft_consensus.cc:2781] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:47.600224 7911 raft_consensus.cc:686] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 1 LEADER]: Becoming Leader. State: Replica: a77f5f40817949d4a40cd63d82b75135, State: Running, Role: LEADER
I20260619 13:07:47.600836 7911 consensus_queue.cc:237] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } }
I20260619 13:07:47.602926 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.602487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:34371)
0619 13:07:47.602681 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.602775 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:07:47.604987 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.604593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:52651)
0619 13:07:47.604771 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.604857 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 13:07:47.605187 7330 catalog_manager.cc:5495] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 reported cstate change: term changed from 0 to 1, leader changed from <none> to a77f5f40817949d4a40cd63d82b75135 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a77f5f40817949d4a40cd63d82b75135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77f5f40817949d4a40cd63d82b75135" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44983 } health_report { overall_health: HEALTHY } } }
I20260619 13:07:47.694770 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.689220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.97.2:44983)
0619 13:07:47.689824 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.689828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.689843 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:07:47.689927 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.689930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.690003 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.690147 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:47.690469 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.691124 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.692450 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.692906 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.693042 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.693462 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:47.694207 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:47.694337 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.694443 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.694517 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":412,"threads_started":1}
I20260619 13:07:47.698765 7466 raft_consensus.cc:1261] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Refusing update from remote peer a77f5f40817949d4a40cd63d82b75135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:47.699050 7755 raft_consensus.cc:1261] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Refusing update from remote peer a77f5f40817949d4a40cd63d82b75135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:47.699466 7919 consensus_queue.cc:1035] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96950338f6c44e0282ba11e5ea2e0289" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:47.699816 7911 consensus_queue.cc:1035] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6246d182494890ad91fb0576ec0233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:47.714901 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.702307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:58983)
0619 13:07:47.714464 (+ 12157us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.714609 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11962,"threads_started":1}
I20260619 13:07:47.716955 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.702051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:40221)
0619 13:07:47.716630 (+ 14579us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.716756 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14425,"thread_start_us":14448,"threads_started":1}
I20260619 13:07:47.723551 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.723141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.3:44915)
0619 13:07:47.723290 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.723385 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 13:07:47.724771 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.724181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.1:33853)
0619 13:07:47.724440 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.724545 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 13:07:47.736217 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.735660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:51395)
0619 13:07:47.735901 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.735974 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 13:07:47.737708 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.737297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.97.2:45529)
0619 13:07:47.737447 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:47.737521 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 13:07:47.739271 7331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:47.739516 7331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:47.746490 7331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb4fcd5fc7594f52b70c699e175048ad: Sending DeleteTablet for 3 replicas of tablet 8777f968de124d0f860c05d8e6bb1241
I20260619 13:07:47.747941 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet 8777f968de124d0f860c05d8e6bb1241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35398
I20260619 13:07:47.748070 7735 tablet_service.cc:1514] Processing DeleteTablet for tablet 8777f968de124d0f860c05d8e6bb1241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:41676
I20260619 13:07:47.748128 7734 tablet_service.cc:1514] Processing DeleteTablet for tablet 8777f968de124d0f860c05d8e6bb1241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:41676
I20260619 13:07:47.749037 7446 tablet_service.cc:1514] Processing DeleteTablet for tablet 8777f968de124d0f860c05d8e6bb1241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46888
I20260619 13:07:47.749886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7333
I20260619 13:07:47.778141 7944 tablet_replica.cc:330] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: stopping tablet replica
I20260619 13:07:47.778479 7944 raft_consensus.cc:2229] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:47.780602 7943 tablet_replica.cc:330] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: stopping tablet replica
I20260619 13:07:47.780872 7943 raft_consensus.cc:2229] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:47.781243 7943 raft_consensus.cc:2258] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:47.781651 7944 raft_consensus.cc:2258] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:47.782742 7944 ts_tablet_manager.cc:1905] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:47.784348 7943 ts_tablet_manager.cc:1905] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:47.787269 7942 tablet_replica.cc:330] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: stopping tablet replica
I20260619 13:07:47.787468 7942 raft_consensus.cc:2229] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:47.788178 7942 raft_consensus.cc:2258] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:47.789251 7942 ts_tablet_manager.cc:1905] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:47.791616 7944 ts_tablet_manager.cc:1918] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:47.791770 7944 log.cc:1196] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/8777f968de124d0f860c05d8e6bb1241
I20260619 13:07:47.792222 7944 ts_tablet_manager.cc:1939] T 8777f968de124d0f860c05d8e6bb1241 P 96950338f6c44e0282ba11e5ea2e0289: Deleting consensus metadata
I20260619 13:07:47.793481 7316 catalog_manager.cc:4841] TS 96950338f6c44e0282ba11e5ea2e0289 (127.0.97.1:32951): tablet 8777f968de124d0f860c05d8e6bb1241 (table test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]) successfully deleted
I20260619 13:07:47.796339 7942 ts_tablet_manager.cc:1918] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:47.796489 7942 log.cc:1196] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/8777f968de124d0f860c05d8e6bb1241
I20260619 13:07:47.796551 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.752142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.97.1:32951)
0619 13:07:47.752384 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.752393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.775404 (+ 23011us) server_negotiation.cc:407] Connection header received
0619 13:07:47.776116 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.776127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.776242 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.776346 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:47.780155 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.781351 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.783554 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.784232 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.784367 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.788341 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:47.788361 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:47.788381 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:47.788430 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:47.792789 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:47.794080 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:47.794092 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:47.794098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:47.794283 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:47.794753 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:47.794758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:47.794762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:47.795236 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:47.795541 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.795909 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.796147 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:07:47.797225 7506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:47.802353 7943 ts_tablet_manager.cc:1918] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:47.802500 7943 log.cc:1196] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/8777f968de124d0f860c05d8e6bb1241
I20260619 13:07:47.802914 7943 ts_tablet_manager.cc:1939] T 8777f968de124d0f860c05d8e6bb1241 P fb6246d182494890ad91fb0576ec0233: Deleting consensus metadata
I20260619 13:07:47.804171 7317 catalog_manager.cc:4841] TS fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925): tablet 8777f968de124d0f860c05d8e6bb1241 (table test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]) successfully deleted
W20260619 13:07:47.804656 7317 catalog_manager.cc:4808] TS fb6246d182494890ad91fb0576ec0233 (127.0.97.3:41925): delete failed for tablet 8777f968de124d0f860c05d8e6bb1241 because the tablet was not found. No further retry: Not found: Tablet not found: 8777f968de124d0f860c05d8e6bb1241
I20260619 13:07:47.804749 7942 ts_tablet_manager.cc:1939] T 8777f968de124d0f860c05d8e6bb1241 P a77f5f40817949d4a40cd63d82b75135: Deleting consensus metadata
I20260619 13:07:47.805944 7315 catalog_manager.cc:4841] TS a77f5f40817949d4a40cd63d82b75135 (127.0.97.2:44983): tablet 8777f968de124d0f860c05d8e6bb1241 (table test-table [id=3241a5f2488d4a99bb799b0ff71d9aaa]) successfully deleted
I20260619 13:07:47.903915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7333
I20260619 13:07:47.921981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7523
I20260619 13:07:47.938249 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:47.922871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.2:44983)
0619 13:07:47.923079 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:47.923086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:47.923108 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:07:47.923263 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:47.923269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:47.923356 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:47.923461 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:47.924777 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.925970 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.927287 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:47.927959 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:47.928081 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:47.930900 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:47.930918 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:47.930939 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:47.930998 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:47.935116 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:47.935875 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:47.935885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:47.935889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:47.936055 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:47.936477 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:47.936483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:47.936486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:47.936940 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:47.937280 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:47.937538 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 13:07:47.937851 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:07:47.939071 7654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:48.057845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7523
I20260619 13:07:48.077018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7672
I20260619 13:07:48.090965 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:48.077902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.97.3:41925)
0619 13:07:48.078123 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:48.078129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:48.078208 (+ 79us) server_negotiation.cc:407] Connection header received
0619 13:07:48.078429 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:48.078434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:48.078575 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:48.078693 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:48.080091 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:48.080815 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:48.082152 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:48.082567 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:48.082727 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:48.085099 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:48.085120 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:48.085143 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:48.085174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:48.088080 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:48.088745 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:48.088755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:48.088758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:48.088886 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:48.089304 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:48.089310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:48.089314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:48.089752 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:48.090061 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:48.090404 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 13:07:48.090630 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:07:48.091683 7795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:48.157725 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:48.156858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53426)
0619 13:07:48.157356 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:48.157457 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":275,"threads_started":1}
I20260619 13:07:48.159194 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:48.158842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37865 (local address 127.0.0.1:53440)
0619 13:07:48.158995 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:48.159074 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 13:07:48.215952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7672
I20260619 13:07:48.238593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7000
I20260619 13:07:48.239755 7167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:48.357929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7000
I20260619 13:07:48.379992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7100
I20260619 13:07:48.381120 7362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:48.501765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7100
2026-06-19T13:07:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12360 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:48 dist-test-slave-q11v krb5kdc[7955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:48 dist-test-slave-q11v krb5kdc[7955](info): set up 2 sockets
Jun 19 13:07:48 dist-test-slave-q11v krb5kdc[7955](info): commencing operation
krb5kdc: starting...
W20260619 13:07:49.638654 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:49 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 13:07:49.853490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40059
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:50.181385 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:50.181777 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:50.181960 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:50.189262 7968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:50.189375 7968 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:07:50.189453 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:50.189548 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:50.189679 7968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:50.189795 7968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:50.199869 7968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:50.201220 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:50.202631 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:07:50.216027 7968 server_base.cc:1034] running on GCE node
W20260619 13:07:50.216048 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:50.216150 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:50.216121 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:50.217366 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:50.218861 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:50.220122 7968 hybrid_clock.cc:648] HybridClock initialized: now 1781874470220044 us; error 59 us; skew 500 ppm
I20260619 13:07:50.220461 7968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:50 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:50.223719 7968 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 13:07:50.225184 7968 webserver.cc:460] Webserver started at http://127.0.97.62:35673/ using document root <none> and password file <none>
I20260619 13:07:50.225986 7968 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:50.226123 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:50.226455 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:50.229724 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "b2f8d661028c44f09540c96d68f8f52f"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "9246949f0107633e985aca8310984dd6"
server_key_iv: "37f17bc3d8a51a67412c15bfaa3d9c54"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.230569 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "b2f8d661028c44f09540c96d68f8f52f"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "9246949f0107633e985aca8310984dd6"
server_key_iv: "37f17bc3d8a51a67412c15bfaa3d9c54"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.235951 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:50.239419 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:50.240780 7968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:07:50.240994 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "b2f8d661028c44f09540c96d68f8f52f"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "9246949f0107633e985aca8310984dd6"
server_key_iv: "37f17bc3d8a51a67412c15bfaa3d9c54"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.241227 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:50.256705 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:50.260697 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:50.260986 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:50.282501 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40059
I20260619 13:07:50.282529 8044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40059 every 8 connection(s)
I20260619 13:07:50.283911 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:50.284814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 7968
I20260619 13:07:50.285017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:50.285844 388 external_mini_cluster.cc:1428] Setting key b86cbeb52b2d4914b270e0a93ab267fc
I20260619 13:07:50.288524 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:50.298933 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Bootstrap starting.
Jun 19 13:07:50 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:50.302166 8045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:50.303248 8045 log.cc:824] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:50.306257 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: No bootstrap required, opened a new log
I20260619 13:07:50.307312 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:50.287775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.97.62:40059)
0619 13:07:50.288491 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:50.288503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:50.288547 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:07:50.290238 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:50.290260 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:50.290637 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:50.291041 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:50.292431 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.293840 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.295125 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.295848 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.296070 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:50.299957 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:50.299965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:50.299980 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:50.300015 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:50.303139 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.304017 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.304024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.304030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.304188 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.304619 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.304625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.304628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.305102 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:50.305481 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:50.305905 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 13:07:50.306345 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":333,"threads_started":1}
I20260619 13:07:50.314172 8045 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } }
I20260619 13:07:50.314566 8045 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:50.314693 8045 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f8d661028c44f09540c96d68f8f52f, State: Initialized, Role: FOLLOWER
I20260619 13:07:50.315232 8045 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } }
I20260619 13:07:50.315475 8045 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:50.315652 8045 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:50.315830 8045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:50.318058 8045 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } }
I20260619 13:07:50.318557 8045 leader_election.cc:302] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f8d661028c44f09540c96d68f8f52f; no voters:
I20260619 13:07:50.319126 8045 leader_election.cc:288] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:50.319274 8050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:50.320004 8050 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [term 1 LEADER]: Becoming Leader. State: Replica: b2f8d661028c44f09540c96d68f8f52f, State: Running, Role: LEADER
I20260619 13:07:50.320528 8050 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } }
I20260619 13:07:50.321202 8045 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:50.323519 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f8d661028c44f09540c96d68f8f52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } } }
I20260619 13:07:50.323947 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:50.324015 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f8d661028c44f09540c96d68f8f52f. Latest consensus state: current_term: 1 leader_uuid: "b2f8d661028c44f09540c96d68f8f52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8d661028c44f09540c96d68f8f52f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40059 } } }
I20260619 13:07:50.324605 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:50.326965 8059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:50.331998 8059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:50.341003 8059 catalog_manager.cc:1347] Generated new cluster ID: abbfc4040f53437692b216c0be80f3f9
I20260619 13:07:50.341123 8059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:50.362502 8059 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:50.364029 8059 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:50.376379 8059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Generated new TSK 0
I20260619 13:07:50.377496 8059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 13:07:50.438602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40059
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:50.769834 8071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:50.770136 8071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:50.770327 8071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:50.777824 8071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:50.777976 8071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:50.778126 8071 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:50.778221 8071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:50.788151 8071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:50.789456 8071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:50.790849 8071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:50.804418 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:50.804379 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:50.804392 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:50.805140 8071 server_base.cc:1034] running on GCE node
I20260619 13:07:50.805768 8071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:50.806785 8071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:50.808053 8071 hybrid_clock.cc:648] HybridClock initialized: now 1781874470807980 us; error 51 us; skew 500 ppm
I20260619 13:07:50.808383 8071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:50 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:50.811741 8071 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:07:50.813182 8071 webserver.cc:460] Webserver started at http://127.0.97.1:46317/ using document root <none> and password file <none>
I20260619 13:07:50.814076 8071 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:50.814209 8071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:50.814532 8071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:50.817432 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "c3d5298705fb4d738223441bb2c37416"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "994d17ee6f4754f1e6aa137727409979"
server_key_iv: "5e5953e48749d16d84be78afb82a9074"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.818310 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c3d5298705fb4d738223441bb2c37416"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "994d17ee6f4754f1e6aa137727409979"
server_key_iv: "5e5953e48749d16d84be78afb82a9074"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.823549 8071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:07:50.827214 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:50.828511 8071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:07:50.828687 8071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "c3d5298705fb4d738223441bb2c37416"
format_stamp: "Formatted at 2026-06-19 13:07:50 on dist-test-slave-q11v"
server_key: "994d17ee6f4754f1e6aa137727409979"
server_key_iv: "5e5953e48749d16d84be78afb82a9074"
server_key_version: "encryptionkey@0"
I20260619 13:07:50.828914 8071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:50.840626 8071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:50.844769 8071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:50.845084 8071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:50.847836 8071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:50.847952 8071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:50.848101 8071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:50.848181 8071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:07:50 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:50.873175 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:50.848543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44039 (local address 127.0.97.62:40059)
0619 13:07:50.849084 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:50.849091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:50.850303 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 13:07:50.852288 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:50.852297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:50.852417 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:50.852540 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:50.854837 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.856021 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.858708 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.859444 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.859611 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:50.865004 (+ 5393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:50.865015 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:50.865023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:50.865081 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:50.869326 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.870285 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.870295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.870301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.870478 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.871099 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.871105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.871108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.871541 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:50.871709 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:50.872552 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 13:07:50.872778 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":303,"threads_started":1}
I20260619 13:07:50.874773 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:50.849230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.1:44039)
0619 13:07:50.850104 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:50.850161 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:50.851989 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:50.852797 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:50.852805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:50.853543 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:50.854601 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:50.854635 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.856218 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.856224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:50.858519 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:50.858539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.860025 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.860037 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:50.860386 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:50.861242 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:50.861268 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:50.864696 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:50.869484 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:50.869494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:50.869515 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:50.870123 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:50.870617 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:50.870626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:50.870632 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:50.870959 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:50.871690 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:50.871699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:50.872144 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 13:07:50.872468 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 13:07:50.873081 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":384,"threads_started":1}
I20260619 13:07:50.891866 8071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33935
I20260619 13:07:50.891867 8206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33935 every 8 connection(s)
I20260619 13:07:50.893564 8071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:50.895841 8207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40059
I20260619 13:07:50.896091 8207 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:50.896929 8207 heartbeater.cc:511] Master 127.0.97.62:40059 requested a full tablet report, sending...
I20260619 13:07:50.898487 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:50.882595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58697 (local address 127.0.97.62:40059)
0619 13:07:50.882802 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:50.882806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:50.883518 (+ 712us) server_negotiation.cc:407] Connection header received
0619 13:07:50.883818 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:50.883824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:50.883903 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:50.884016 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:50.885559 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.886683 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.888916 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.889515 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.889706 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:50.892208 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:50.892215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:50.892220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:50.892261 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:50.895510 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.896416 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.896421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.896425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.896576 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:50.896950 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:50.896955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:50.896959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:50.897298 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:50.897449 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:50.898010 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 13:07:50.898230 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:07:50.898660 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:50.882650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.1:58697)
0619 13:07:50.883391 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:50.883418 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:50.883625 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:50.884049 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:50.884054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:50.884807 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:50.885341 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:50.885361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.886886 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.886890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:50.888725 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:50.888742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:50.889758 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:50.889764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:50.889941 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:50.890645 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:50.890672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:50.891980 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:50.895679 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:50.895684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:50.895688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:50.896303 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:50.896692 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:50.896695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:50.896697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:50.896838 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:50.897469 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:50.897473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:50.897703 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 13:07:50.897887 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 13:07:50.898212 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":338,"threads_started":1}
I20260619 13:07:50.899426 8000 ts_manager.cc:194] Registered new tserver with Master: c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935)
I20260619 13:07:50.900626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8071
I20260619 13:07:50.900826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:50.901021 8000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44039
I20260619 13:07:50.901636 388 external_mini_cluster.cc:1428] Setting key b3673dc4456d7edbcc80395d0d6ab353
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 13:07:50.946935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40059
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:51.274777 8211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:51.275134 8211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:51.275324 8211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:51.282954 8211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:51.283113 8211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:51.283270 8211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:51.283413 8211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:51.293211 8211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:51.294572 8211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:51.295969 8211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:51.309477 8219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:51.309497 8217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:51.309577 8216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:51.310141 8211 server_base.cc:1034] running on GCE node
I20260619 13:07:51.310835 8211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:51.311807 8211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:51.313033 8211 hybrid_clock.cc:648] HybridClock initialized: now 1781874471312956 us; error 46 us; skew 500 ppm
I20260619 13:07:51.313359 8211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:51 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:51.316586 8211 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:07:51.318156 8211 webserver.cc:460] Webserver started at http://127.0.97.2:43659/ using document root <none> and password file <none>
I20260619 13:07:51.318833 8211 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:51.318918 8211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:51.319173 8211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:51.322142 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "3684f3b2841e4014907151033a2c01d7"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "eb604bca5ab0bf492464aef509750e0b"
server_key_iv: "bf28360a30a93efc8fedfbe515e4b607"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.322968 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3684f3b2841e4014907151033a2c01d7"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "eb604bca5ab0bf492464aef509750e0b"
server_key_iv: "bf28360a30a93efc8fedfbe515e4b607"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.328248 8211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:07:51.331638 8226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:51.333204 8211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:51.333406 8211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "3684f3b2841e4014907151033a2c01d7"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "eb604bca5ab0bf492464aef509750e0b"
server_key_iv: "bf28360a30a93efc8fedfbe515e4b607"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.333710 8211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:51.358966 8211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:51.363127 8211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:51.363430 8211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:51.366189 8211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:51.366314 8211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:51.366477 8211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:51.366562 8211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:07:51 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:51.388811 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.366968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58351 (local address 127.0.97.62:40059)
0619 13:07:51.367207 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:51.367217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:51.368607 (+ 1390us) server_negotiation.cc:407] Connection header received
0619 13:07:51.370676 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:51.370685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:51.370792 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:51.370929 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:51.373338 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.374444 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.376165 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.376607 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.376785 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.381525 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:51.381530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:51.381533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:51.381567 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:51.385146 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.386378 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.386386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.386389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.386530 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.387002 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.387007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.387011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.387296 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:51.387466 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:51.388228 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 13:07:51.388464 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:07:51.390202 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.367519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.2:58351)
0619 13:07:51.368404 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:51.368450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:51.370398 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:51.371227 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:51.371237 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:51.372019 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:51.373106 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.373144 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.374603 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.374608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.375995 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.376006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.376867 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.376894 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.377154 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.378057 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:51.378089 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:51.381215 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:51.385387 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.385402 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.385429 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.386228 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.386652 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.386658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.386662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.386869 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.387444 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:51.387453 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:51.387831 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 13:07:51.388111 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 13:07:51.388678 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":385,"threads_started":1}
I20260619 13:07:51.421130 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.404306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50515 (local address 127.0.97.62:40059)
0619 13:07:51.404568 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:51.404576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:51.405107 (+ 531us) server_negotiation.cc:407] Connection header received
0619 13:07:51.405273 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:51.405281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:51.405386 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:51.405525 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:51.407317 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.408641 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.410719 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.411329 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.411461 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.414282 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:51.414291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:51.414297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:51.414344 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:51.417512 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.418551 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.418559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.418565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.418747 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.419255 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.419260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.419263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.419736 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:51.419878 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:51.420553 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 13:07:51.420781 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:51.421296 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.404280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.2:50515)
0619 13:07:51.404992 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:51.405019 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:51.405142 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:51.405560 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:51.405564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:51.406489 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:51.406996 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.407019 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.408828 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.408836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.410456 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.410488 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.411564 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.411574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.411823 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.412582 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:51.412616 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:51.414044 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:51.417707 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.417715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.417718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.418383 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.418879 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.418884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.418888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.419083 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.419932 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:51.419936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:51.420232 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 13:07:51.420429 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 13:07:51.420803 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":402,"threads_started":1}
I20260619 13:07:51.430701 8211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45753
I20260619 13:07:51.430728 8345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45753 every 8 connection(s)
I20260619 13:07:51.432034 8211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:51.434242 8346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40059
I20260619 13:07:51.434486 8346 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:51.435146 8346 heartbeater.cc:511] Master 127.0.97.62:40059 requested a full tablet report, sending...
I20260619 13:07:51.436545 8000 ts_manager.cc:194] Registered new tserver with Master: 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753)
I20260619 13:07:51.437633 8000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58351
I20260619 13:07:51.438701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8211
I20260619 13:07:51.438889 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:51.439321 388 external_mini_cluster.cc:1428] Setting key c14a61e0709a95630e4e84df235f2421
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:07:51.485395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40059
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:51.812055 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:51.812400 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:51.812577 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:51.823994 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:51.824146 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:51.824286 8350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:07:51.824429 8350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:51.835388 8350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:51.836802 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:51.838478 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:51.852416 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:51.852476 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:51.852444 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:51.853230 8350 server_base.cc:1034] running on GCE node
I20260619 13:07:51.853941 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:51.854980 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:51.856254 8350 hybrid_clock.cc:648] HybridClock initialized: now 1781874471856187 us; error 39 us; skew 500 ppm
I20260619 13:07:51.856571 8350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:51 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:51.859881 8350 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:07:51.861491 8350 webserver.cc:460] Webserver started at http://127.0.97.3:46451/ using document root <none> and password file <none>
I20260619 13:07:51.862255 8350 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:51.862425 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:51.862782 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:51.865875 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "6c0511c787ca1e219f94bfd5ba796ce1"
server_key_iv: "f85a55805efc16610fedb1341b686f91"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.866714 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "6c0511c787ca1e219f94bfd5ba796ce1"
server_key_iv: "f85a55805efc16610fedb1341b686f91"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.872676 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:07:51.876550 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:51.878100 8350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:51.878408 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
format_stamp: "Formatted at 2026-06-19 13:07:51 on dist-test-slave-q11v"
server_key: "6c0511c787ca1e219f94bfd5ba796ce1"
server_key_iv: "f85a55805efc16610fedb1341b686f91"
server_key_version: "encryptionkey@0"
I20260619 13:07:51.878650 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:51.895702 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:51.900944 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:51.901172 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:51.903752 8350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:51.903914 8350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:51.904049 8350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:51.904114 8350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:51.904172 8207 heartbeater.cc:503] Master 127.0.97.62:40059 was elected leader, sending a full tablet report...
Jun 19 13:07:51 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:51.931072 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.904543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48707 (local address 127.0.97.62:40059)
0619 13:07:51.904775 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:51.904784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:51.906592 (+ 1808us) server_negotiation.cc:407] Connection header received
0619 13:07:51.908845 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:51.908851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:51.908964 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:51.909097 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:51.912090 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.913091 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.916597 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.917191 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.917347 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.923322 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:51.923330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:51.923335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:51.923376 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:51.926995 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.928052 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.928057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.928061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.928194 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.928840 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.928846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.928850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.929246 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:51.929435 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:51.930472 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 13:07:51.930703 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:07:51.932865 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.905513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.3:48707)
0619 13:07:51.906432 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:51.906478 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:51.908473 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:51.909628 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:51.909650 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:51.910642 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:51.911819 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.911864 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.913320 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.913329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.916387 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.916413 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.917484 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.917498 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.917900 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.918828 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:51.918868 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:51.922929 (+ 4061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:51.927198 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.927212 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.927231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.927907 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.928384 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.928392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.928398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.928699 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.929388 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:51.929399 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:51.929954 (+ 555us) client_negotiation.cc:769] Sending connection context
0619 13:07:51.930371 (+ 417us) client_negotiation.cc:240] Negotiation successful
0619 13:07:51.930969 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":424,"threads_started":1}
I20260619 13:07:51.946697 8350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43851
I20260619 13:07:51.946678 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43851 every 8 connection(s)
I20260619 13:07:51.948078 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:51.950445 8485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40059
I20260619 13:07:51.950686 8485 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:51.951376 8485 heartbeater.cc:511] Master 127.0.97.62:40059 requested a full tablet report, sending...
I20260619 13:07:51.952775 8000 ts_manager.cc:194] Registered new tserver with Master: fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:51.953743 8000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48707
I20260619 13:07:51.955308 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.939873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48057 (local address 127.0.97.62:40059)
0619 13:07:51.940052 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:51.940056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:51.941198 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 13:07:51.941421 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:51.941426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:51.941511 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:51.941659 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:51.943309 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.944584 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.946623 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.947090 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.947248 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.949376 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:51.949383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:51.949390 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:51.949429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:51.952464 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.953201 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.953207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.953211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.953352 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:51.953778 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:51.953784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:51.953788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:51.954123 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:51.954295 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:51.954714 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 13:07:51.954935 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:07:51.955462 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:51.940084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.97.3:48057)
0619 13:07:51.941086 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:51.941107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:51.941278 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:51.941869 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:51.941878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:51.942579 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:51.943076 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.943096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.944730 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.944739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.946432 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:51.946451 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:51.947296 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:51.947303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:51.947479 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:51.948145 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:51.948168 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:51.949150 (+ 982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:51.952607 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.952614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.952618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.953065 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.953472 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:51.953478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:51.953481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:51.953665 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:51.954259 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:51.954264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:51.954456 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 13:07:51.954602 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 13:07:51.954941 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":374,"threads_started":1}
I20260619 13:07:51.956926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8350
I20260619 13:07:51.957104 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:51.957576 388 external_mini_cluster.cc:1428] Setting key 462f3bedade0340bb5be95ff905346cb
I20260619 13:07:51.960503 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:07:51 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:07:52.010475 8095 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:51.999867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.97.62:40059)
0619 13:07:52.000089 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.000096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.000353 (+ 257us) server_negotiation.cc:407] Connection header received
0619 13:07:52.000535 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.000540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.000611 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.000695 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:52.002024 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.002708 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.003975 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.004422 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.004524 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.010115 (+ 5591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42702: BlockingRecv error: recv got EOF from 127.0.0.1:42702 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:52.028592 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.013893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.97.62:40059)
0619 13:07:52.014097 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.014103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.014312 (+ 209us) server_negotiation.cc:407] Connection header received
0619 13:07:52.014436 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.014439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.014528 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.014630 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:52.015906 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.016764 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.018310 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.018740 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.018848 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.022814 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:52.022819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:52.022823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:52.022858 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:52.025683 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.026508 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.026513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.026516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.026700 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.027152 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.027156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.027158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.027530 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:52.027776 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.028062 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.028246 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:07:52.032987 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:52.041989 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:52.069334 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.055788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.0.97.3:43851)
0619 13:07:52.056919 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.056929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.057811 (+ 882us) server_negotiation.cc:407] Connection header received
0619 13:07:52.058308 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.058321 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.058613 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.058781 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.059984 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.062150 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.065743 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.067492 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.067613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.067939 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.068745 (+ 806us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.068965 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":773,"threads_started":1}
I20260619 13:07:52.070168 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.055525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43851 (local address 127.0.0.1:48852)
0619 13:07:52.057622 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.057674 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.058008 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.059167 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.059174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.059204 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.059676 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.059694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.062348 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.062353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.065469 (+ 3116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.065491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.068141 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.068151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.068447 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.068514 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.069562 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.069790 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":1566,"threads_started":1}
I20260619 13:07:52.074373 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdcd38feb94b56912bf365d50b7ced (DEFAULT_TABLE table=test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.076071 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdcd38feb94b56912bf365d50b7ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.076427 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.056980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.0.97.2:45753)
0619 13:07:52.068269 (+ 11289us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.068281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.068317 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:07:52.068421 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.068430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.068832 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.069676 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.069906 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.071352 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.074026 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.075323 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.075488 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.075706 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.075960 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.076121 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11087,"thread_start_us":10937,"threads_started":1}
I20260619 13:07:52.076738 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.058792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33935 (local address 127.0.0.1:40950)
0619 13:07:52.068585 (+ 9793us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.068603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.068709 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.069906 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.069911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.069941 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.070350 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.070364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.072022 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.072028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.074151 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.074160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.075887 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.075893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.076082 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.076126 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.076243 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.076374 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9679,"mutex_wait_us":25,"thread_start_us":9517,"threads_started":1}
I20260619 13:07:52.077011 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.056367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.0.1:50052)
0619 13:07:52.067287 (+ 10920us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.067312 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.067469 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.069079 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.069083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.069110 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.069407 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.069455 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.071516 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.071521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.073859 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.073873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.075553 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.075559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.075719 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.075767 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.075898 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.076051 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10740,"mutex_wait_us":72,"thread_start_us":11199,"threads_started":1}
I20260619 13:07:52.077265 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.058445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.0.97.1:33935)
0619 13:07:52.069188 (+ 10743us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.069196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.069223 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:07:52.069317 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.069327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.069686 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.069895 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.070552 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.071861 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.074299 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.075358 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.075523 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.075729 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.076749 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.076933 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10581,"thread_start_us":10444,"threads_started":1}
I20260619 13:07:52.080729 8137 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdcd38feb94b56912bf365d50b7ced (DEFAULT_TABLE table=test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.080729 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdcd38feb94b56912bf365d50b7ced (DEFAULT_TABLE table=test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.082170 8137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdcd38feb94b56912bf365d50b7ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.082183 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdcd38feb94b56912bf365d50b7ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.088748 8509 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Bootstrap starting.
I20260619 13:07:52.091995 8509 tablet_bootstrap.cc:654] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.093412 8509 log.cc:824] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:52.097945 8512 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Bootstrap starting.
I20260619 13:07:52.098269 8509 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: No bootstrap required, opened a new log
I20260619 13:07:52.098555 8509 ts_tablet_manager.cc:1397] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:07:52.100757 8512 tablet_bootstrap.cc:654] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.101233 8511 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Bootstrap starting.
I20260619 13:07:52.101905 8512 log.cc:824] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:52.104375 8512 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: No bootstrap required, opened a new log
I20260619 13:07:52.104626 8512 ts_tablet_manager.cc:1397] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:07:52.104673 8511 tablet_bootstrap.cc:654] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.105908 8511 log.cc:824] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Log is configured to *not* fsync() on all Append() calls
I20260619 13:07:52.108039 8509 raft_consensus.cc:348] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.108522 8509 raft_consensus.cc:374] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.108628 8509 raft_consensus.cc:729] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.109225 8511 tablet_bootstrap.cc:492] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: No bootstrap required, opened a new log
I20260619 13:07:52.109304 8509 consensus_queue.cc:260] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.109503 8511 ts_tablet_manager.cc:1397] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:07:52.110461 8485 heartbeater.cc:503] Master 127.0.97.62:40059 was elected leader, sending a full tablet report...
I20260619 13:07:52.111143 8509 ts_tablet_manager.cc:1428] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 13:07:52.114187 8512 raft_consensus.cc:348] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.114667 8512 raft_consensus.cc:374] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.114773 8512 raft_consensus.cc:729] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.115411 8512 consensus_queue.cc:260] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.116475 8346 heartbeater.cc:503] Master 127.0.97.62:40059 was elected leader, sending a full tablet report...
I20260619 13:07:52.117201 8512 ts_tablet_manager.cc:1428] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:07:52.120085 8511 raft_consensus.cc:348] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.120544 8511 raft_consensus.cc:374] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.120654 8511 raft_consensus.cc:729] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.121266 8511 consensus_queue.cc:260] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.122960 8511 ts_tablet_manager.cc:1428] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
W20260619 13:07:52.146309 8208 tablet.cc:2369] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:52.148800 8516 raft_consensus.cc:482] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:52.149166 8516 raft_consensus.cc:504] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.151734 8516 leader_election.cc:288] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851), c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935)
I20260619 13:07:52.160966 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.153844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52529 (local address 127.0.97.1:33935)
0619 13:07:52.154091 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.154097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.154114 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:52.154179 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.154183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.154280 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.154366 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.155149 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.156181 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.158776 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.159652 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.159792 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.159923 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.160573 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.160711 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 13:07:52.161291 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.152795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33935 (local address 127.0.97.2:52529)
0619 13:07:52.153380 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.153399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.153534 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.154604 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.154609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.154632 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.154967 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.154978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.156372 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.156376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.158617 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.158629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.160066 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.160072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.160294 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.160349 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.160514 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.160658 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":61,"thread_start_us":319,"threads_started":1}
I20260619 13:07:52.161352 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.152151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40541 (local address 127.0.97.3:43851)
0619 13:07:52.152411 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.152418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.152704 (+ 286us) server_negotiation.cc:407] Connection header received
0619 13:07:52.152836 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.152843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.152922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.153022 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.154091 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.155496 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.158593 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.160025 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.160136 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.160251 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.160940 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.161088 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 13:07:52.162174 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.151968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43851 (local address 127.0.97.2:40541)
0619 13:07:52.152519 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.152537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.152688 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.153223 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.153231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.153260 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.153746 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.153774 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.155685 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.155690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.158387 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.158405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.160499 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.160507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.160723 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.160760 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.161635 (+ 875us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.161792 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":31,"thread_start_us":452,"threads_started":1}
I20260619 13:07:52.162648 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdcd38feb94b56912bf365d50b7ced" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416" is_pre_election: true
I20260619 13:07:52.163240 8157 raft_consensus.cc:2454] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:52.163969 8230 leader_election.cc:302] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416; no voters:
I20260619 13:07:52.164469 8516 raft_consensus.cc:2781] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:52.164657 8516 raft_consensus.cc:482] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:52.164510 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdcd38feb94b56912bf365d50b7ced" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" is_pre_election: true
I20260619 13:07:52.164768 8516 raft_consensus.cc:3037] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.165058 8435 raft_consensus.cc:2454] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:52.167425 8516 raft_consensus.cc:504] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.167973 8516 leader_election.cc:288] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Requested vote from peers fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851), c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935)
I20260619 13:07:52.168490 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdcd38feb94b56912bf365d50b7ced" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
I20260619 13:07:52.168720 8435 raft_consensus.cc:3037] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.168704 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdcd38feb94b56912bf365d50b7ced" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416"
I20260619 13:07:52.168919 8157 raft_consensus.cc:3037] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.172029 8157 raft_consensus.cc:2454] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:52.172331 8435 raft_consensus.cc:2454] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:52.172451 8230 leader_election.cc:302] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416; no voters:
I20260619 13:07:52.172780 8516 raft_consensus.cc:2781] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:52.173322 8516 raft_consensus.cc:686] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Running, Role: LEADER
I20260619 13:07:52.174005 8516 consensus_queue.cc:237] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } }
I20260619 13:07:52.179579 7999 catalog_manager.cc:5495] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3684f3b2841e4014907151033a2c01d7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } health_report { overall_health: HEALTHY } } }
W20260619 13:07:52.184123 8347 tablet.cc:2369] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:07:52.199899 8486 tablet.cc:2369] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:07:52.221273 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.216017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.97.2:45753)
0619 13:07:52.216240 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.216246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.216265 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:07:52.216460 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.216464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.216546 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.216705 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:52.217074 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.217886 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.219195 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.219580 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.219719 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.220162 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.220698 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.220861 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.220916 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.220992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:07:52.229240 8157 raft_consensus.cc:1261] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.229241 8435 raft_consensus.cc:1261] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.230262 8520 consensus_queue.cc:1035] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.230623 8516 consensus_queue.cc:1035] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.242151 8523 mvcc.cc:205] Tried to move back new op lower bound from 7298557838241959936 to 7298557838030450688. Current Snapshot: MvccSnapshot[applied={T|T < 7298557838241959936}]
I20260619 13:07:52.247586 8525 mvcc.cc:205] Tried to move back new op lower bound from 7298557838241959936 to 7298557838030450688. Current Snapshot: MvccSnapshot[applied={T|T < 7298557838241959936}]
I20260619 13:07:52.256502 8000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:52.256789 8000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:52.260004 8000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Sending DeleteTablet for 3 replicas of tablet 1abdcd38feb94b56912bf365d50b7ced
I20260619 13:07:52.261092 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdcd38feb94b56912bf365d50b7ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48852
I20260619 13:07:52.261168 8137 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdcd38feb94b56912bf365d50b7ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 13:07:52.261353 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdcd38feb94b56912bf365d50b7ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50052
I20260619 13:07:52.263633 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:52.264431 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:52.276536 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 (DEFAULT_TABLE table=test-table [id=3710ce799a3d42458854422d5f5900cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.276983 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 (DEFAULT_TABLE table=test-table [id=3710ce799a3d42458854422d5f5900cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.277127 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daa40f3aa814c528811d20a87b3aa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.277542 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daa40f3aa814c528811d20a87b3aa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.286249 8275 tablet_service.cc:1467] Processing CreateTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 (DEFAULT_TABLE table=test-table [id=3710ce799a3d42458854422d5f5900cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.286872 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daa40f3aa814c528811d20a87b3aa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.291406 8509 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Bootstrap starting.
I20260619 13:07:52.293709 8509 tablet_bootstrap.cc:654] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.299294 8535 tablet_replica.cc:330] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: stopping tablet replica
I20260619 13:07:52.300802 8534 tablet_replica.cc:330] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: stopping tablet replica
I20260619 13:07:52.301286 8534 raft_consensus.cc:2229] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:52.301800 8534 raft_consensus.cc:2258] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.302482 8535 raft_consensus.cc:2229] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:52.303078 8535 raft_consensus.cc:2258] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.303809 8511 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Bootstrap starting.
I20260619 13:07:52.304206 8512 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Bootstrap starting.
I20260619 13:07:52.305869 8511 tablet_bootstrap.cc:654] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.306473 8512 tablet_bootstrap.cc:654] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.311527 8533 tablet_replica.cc:330] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: stopping tablet replica
I20260619 13:07:52.312052 8533 raft_consensus.cc:2229] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:52.312724 8533 raft_consensus.cc:2258] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.313740 8535 ts_tablet_manager.cc:1905] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.316391 8511 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: No bootstrap required, opened a new log
I20260619 13:07:52.316532 8511 ts_tablet_manager.cc:1397] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260619 13:07:52.317298 8511 raft_consensus.cc:348] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.317528 8511 raft_consensus.cc:374] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.317648 8511 raft_consensus.cc:729] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.317971 8511 consensus_queue.cc:260] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.318373 8533 ts_tablet_manager.cc:1905] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.318712 8509 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: No bootstrap required, opened a new log
I20260619 13:07:52.318871 8509 ts_tablet_manager.cc:1397] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20260619 13:07:52.319492 8534 ts_tablet_manager.cc:1905] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.319710 8509 raft_consensus.cc:348] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.319940 8509 raft_consensus.cc:374] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.320057 8509 raft_consensus.cc:729] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.320333 8509 consensus_queue.cc:260] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.320928 8509 ts_tablet_manager.cc:1428] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:07:52.320960 8511 ts_tablet_manager.cc:1428] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 13:07:52.328364 8533 ts_tablet_manager.cc:1918] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:52.328540 8533 log.cc:1196] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/1abdcd38feb94b56912bf365d50b7ced
I20260619 13:07:52.329067 8533 ts_tablet_manager.cc:1939] T 1abdcd38feb94b56912bf365d50b7ced P fe7eb1331beb4fe788f24db3c0b9211f: Deleting consensus metadata
I20260619 13:07:52.330909 7987 catalog_manager.cc:4841] TS fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851): tablet 1abdcd38feb94b56912bf365d50b7ced (table test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]) successfully deleted
I20260619 13:07:52.332845 8512 tablet_bootstrap.cc:492] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: No bootstrap required, opened a new log
I20260619 13:07:52.334417 8512 ts_tablet_manager.cc:1397] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.007s
I20260619 13:07:52.335335 8535 ts_tablet_manager.cc:1918] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:52.335525 8535 log.cc:1196] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/1abdcd38feb94b56912bf365d50b7ced
I20260619 13:07:52.335417 8512 raft_consensus.cc:348] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.335851 8512 raft_consensus.cc:374] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.335994 8534 ts_tablet_manager.cc:1918] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:52.335999 8512 raft_consensus.cc:729] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.336098 8535 ts_tablet_manager.cc:1939] T 1abdcd38feb94b56912bf365d50b7ced P 3684f3b2841e4014907151033a2c01d7: Deleting consensus metadata
I20260619 13:07:52.336196 8534 log.cc:1196] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/1abdcd38feb94b56912bf365d50b7ced
I20260619 13:07:52.336450 8512 consensus_queue.cc:260] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.336748 8534 ts_tablet_manager.cc:1939] T 1abdcd38feb94b56912bf365d50b7ced P c3d5298705fb4d738223441bb2c37416: Deleting consensus metadata
I20260619 13:07:52.337800 8512 ts_tablet_manager.cc:1428] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:52.339043 7987 catalog_manager.cc:4841] TS c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935): tablet 1abdcd38feb94b56912bf365d50b7ced (table test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]) successfully deleted
I20260619 13:07:52.340039 7984 catalog_manager.cc:4841] TS 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753): tablet 1abdcd38feb94b56912bf365d50b7ced (table test-table [id=866391c57b0f4fc081a3c7b09ef2ad7f]) successfully deleted
I20260619 13:07:52.345701 8515 raft_consensus.cc:482] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:52.345947 8515 raft_consensus.cc:504] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.347520 8515 leader_election.cc:288] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753)
I20260619 13:07:52.354825 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.348435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49635 (local address 127.0.97.2:45753)
0619 13:07:52.348606 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.348612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.348632 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:07:52.348811 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.348816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.348884 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.348947 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.349588 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.350641 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.352379 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.353687 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.353853 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.353953 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.354345 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.354499 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:07:52.354907 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.347713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33935 (local address 127.0.97.3:53175)
0619 13:07:52.348018 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.348038 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.348157 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.348706 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.348711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.348738 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.348949 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.348960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.350439 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.350445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.352465 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.352482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.354102 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.354108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.354345 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.354379 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.354464 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.354592 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":4}
I20260619 13:07:52.355062 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.347896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.97.3:49635)
0619 13:07:52.348528 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.348549 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.348640 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.349139 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.349142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.349161 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:52.349423 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.349429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.350796 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.350802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.352213 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.352224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.353916 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.353923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.354102 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.354197 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.354299 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.354440 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":27,"thread_start_us":773,"threads_started":1}
I20260619 13:07:52.355494 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.347919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53175 (local address 127.0.97.1:33935)
0619 13:07:52.348112 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.348118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.348143 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:07:52.348369 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.348374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.348449 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.348532 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:52.349110 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.350270 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.352631 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.353944 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.354066 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.354161 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.355068 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.355207 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 13:07:52.356101 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daa40f3aa814c528811d20a87b3aa18" candidate_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416" is_pre_election: true
I20260619 13:07:52.356094 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daa40f3aa814c528811d20a87b3aa18" candidate_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3684f3b2841e4014907151033a2c01d7" is_pre_election: true
I20260619 13:07:52.356353 8157 raft_consensus.cc:2454] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe7eb1331beb4fe788f24db3c0b9211f in term 0.
I20260619 13:07:52.356557 8301 raft_consensus.cc:2454] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe7eb1331beb4fe788f24db3c0b9211f in term 0.
I20260619 13:07:52.356854 8369 leader_election.cc:302] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d5298705fb4d738223441bb2c37416, fe7eb1331beb4fe788f24db3c0b9211f; no voters:
I20260619 13:07:52.357277 8515 raft_consensus.cc:2781] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:52.357412 8515 raft_consensus.cc:482] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:52.357509 8515 raft_consensus.cc:3037] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.359588 8515 raft_consensus.cc:504] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.360194 8515 leader_election.cc:288] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [CANDIDATE]: Term 1 election: Requested vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753)
I20260619 13:07:52.360685 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daa40f3aa814c528811d20a87b3aa18" candidate_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416"
I20260619 13:07:52.360760 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daa40f3aa814c528811d20a87b3aa18" candidate_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3684f3b2841e4014907151033a2c01d7"
I20260619 13:07:52.360885 8157 raft_consensus.cc:3037] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.360965 8301 raft_consensus.cc:3037] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.362921 8157 raft_consensus.cc:2454] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe7eb1331beb4fe788f24db3c0b9211f in term 1.
I20260619 13:07:52.363318 8369 leader_election.cc:302] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d5298705fb4d738223441bb2c37416, fe7eb1331beb4fe788f24db3c0b9211f; no voters:
I20260619 13:07:52.363605 8515 raft_consensus.cc:2781] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:52.364086 8515 raft_consensus.cc:686] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 LEADER]: Becoming Leader. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Running, Role: LEADER
I20260619 13:07:52.364195 8301 raft_consensus.cc:2454] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe7eb1331beb4fe788f24db3c0b9211f in term 1.
I20260619 13:07:52.364761 8515 consensus_queue.cc:237] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.368407 7999 catalog_manager.cc:5495] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f reported cstate change: term changed from 0 to 1, leader changed from <none> to fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } health_report { overall_health: HEALTHY } } }
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:52.419118 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.404074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40059 (local address 127.0.0.1:42722)
0619 13:07:52.404768 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.404794 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.404920 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.405288 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.405291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.406033 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:52.406303 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.406311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.407889 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.407893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.408849 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.408858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.409788 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.409793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.409900 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.410617 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:52.410650 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:52.413014 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:52.416748 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:52.416757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:52.416765 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:52.417162 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:52.417555 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:52.417562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:52.417568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:52.417747 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:52.418333 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:52.418339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:52.418482 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.418620 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.418808 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":394,"threads_started":1}
I20260619 13:07:52.419421 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.404074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.97.62:40059)
0619 13:07:52.404325 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.404332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.404885 (+ 553us) server_negotiation.cc:407] Connection header received
0619 13:07:52.405033 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.405041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.405128 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.405248 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:52.406442 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.407655 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.408981 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.409624 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.409747 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.413232 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:52.413238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:52.413243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:52.413307 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:52.416623 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.417277 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.417283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.417287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.417431 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.417865 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.417870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.417874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.418209 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:52.418352 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.418751 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.418963 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 13:07:52.424546 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42722:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:07:52.425575 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:52.442622 8137 tablet_service.cc:1467] Processing CreateTablet for tablet 42f98b74fffc43feb0590b30e1a69af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bade82055b649acbf4bb6ff85e2ef02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:52.442986 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 42f98b74fffc43feb0590b30e1a69af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bade82055b649acbf4bb6ff85e2ef02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:52.443267 8137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f98b74fffc43feb0590b30e1a69af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.443104 8275 tablet_service.cc:1467] Processing CreateTablet for tablet 42f98b74fffc43feb0590b30e1a69af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bade82055b649acbf4bb6ff85e2ef02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:07:52.443507 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f98b74fffc43feb0590b30e1a69af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.443643 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f98b74fffc43feb0590b30e1a69af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.450788 8511 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: Bootstrap starting.
I20260619 13:07:52.450783 8509 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: Bootstrap starting.
I20260619 13:07:52.452488 8511 tablet_bootstrap.cc:654] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.452483 8509 tablet_bootstrap.cc:654] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.453619 8512 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: Bootstrap starting.
I20260619 13:07:52.455924 8512 tablet_bootstrap.cc:654] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.462182 8512 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: No bootstrap required, opened a new log
I20260619 13:07:52.462420 8512 ts_tablet_manager.cc:1397] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260619 13:07:52.463485 8512 raft_consensus.cc:348] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.463828 8512 raft_consensus.cc:374] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.463932 8512 raft_consensus.cc:729] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.464283 8512 consensus_queue.cc:260] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.465375 8529 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.465848 8529 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.469431 8512 ts_tablet_manager.cc:1428] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260619 13:07:52.479425 8509 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: No bootstrap required, opened a new log
I20260619 13:07:52.479622 8509 ts_tablet_manager.cc:1397] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.018s
I20260619 13:07:52.479837 8511 tablet_bootstrap.cc:492] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: No bootstrap required, opened a new log
I20260619 13:07:52.480001 8511 ts_tablet_manager.cc:1397] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.007s
I20260619 13:07:52.480474 8509 raft_consensus.cc:348] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.480724 8509 raft_consensus.cc:374] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.480803 8509 raft_consensus.cc:729] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.480675 8511 raft_consensus.cc:348] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.480926 8511 raft_consensus.cc:374] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.481016 8511 raft_consensus.cc:729] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.481084 8509 consensus_queue.cc:260] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.481251 8511 consensus_queue.cc:260] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.481699 8509 ts_tablet_manager.cc:1428] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 13:07:52.481572 8515 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.481909 8515 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.482280 8517 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.482648 8517 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.483568 8511 ts_tablet_manager.cc:1428] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 13:07:52.609977 8529 raft_consensus.cc:482] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:52.610292 8529 raft_consensus.cc:504] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.611181 8529 leader_election.cc:288] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:52.611771 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f98b74fffc43feb0590b30e1a69af4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416" is_pre_election: true
I20260619 13:07:52.611928 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f98b74fffc43feb0590b30e1a69af4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" is_pre_election: true
I20260619 13:07:52.612061 8157 raft_consensus.cc:2454] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:52.612200 8435 raft_consensus.cc:2454] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:52.612545 8230 leader_election.cc:302] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416; no voters:
I20260619 13:07:52.612811 8529 raft_consensus.cc:2781] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:52.612941 8529 raft_consensus.cc:482] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:52.613054 8529 raft_consensus.cc:3037] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.616438 8529 raft_consensus.cc:504] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.617076 8529 leader_election.cc:288] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Requested vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:52.617489 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f98b74fffc43feb0590b30e1a69af4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416"
I20260619 13:07:52.617661 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f98b74fffc43feb0590b30e1a69af4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
I20260619 13:07:52.617769 8157 raft_consensus.cc:3037] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.617864 8435 raft_consensus.cc:3037] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:52.620684 8435 raft_consensus.cc:2454] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:52.620671 8157 raft_consensus.cc:2454] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:52.621148 8230 leader_election.cc:302] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416; no voters:
I20260619 13:07:52.621577 8529 raft_consensus.cc:2781] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:52.621781 8529 raft_consensus.cc:686] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Running, Role: LEADER
I20260619 13:07:52.622161 8529 consensus_queue.cc:237] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.623837 8516 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: TxnStatusTablet state changed. Reason: New leader 3684f3b2841e4014907151033a2c01d7. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.624156 8516 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:52.624922 8552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:07:52.625223 8552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:07:52.626149 7999 catalog_manager.cc:5495] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3684f3b2841e4014907151033a2c01d7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:52.649880 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.643999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.97.2:45753)
0619 13:07:52.644199 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.644206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.644223 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:07:52.644486 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.644490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.644560 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.644635 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:52.645053 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.645944 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.647862 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.648587 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.648680 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.649210 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.649303 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.649366 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.649547 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.649667 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 13:07:52.650009 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.643865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.0.1:50072)
0619 13:07:52.644130 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.644154 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.644301 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.644700 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.644706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.644720 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:52.644923 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.644935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.646063 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.646069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.647703 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.647721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.648752 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.648758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.648948 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.649022 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.649408 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.649428 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.649523 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.649705 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 13:07:52.653823 8435 raft_consensus.cc:1261] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.653844 8157 raft_consensus.cc:1261] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.654460 8516 consensus_queue.cc:1035] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.654940 8529 consensus_queue.cc:1035] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.658171 8515 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: TxnStatusTablet state changed. Reason: New leader 3684f3b2841e4014907151033a2c01d7. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.658447 8515 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.658944 8517 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: TxnStatusTablet state changed. Reason: New leader 3684f3b2841e4014907151033a2c01d7. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.659228 8517 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.660701 8520 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.661046 8520 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:52.662155 8516 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.662468 8516 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P 3684f3b2841e4014907151033a2c01d7: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:07:52.663632 8517 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.663846 8517 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P c3d5298705fb4d738223441bb2c37416: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.666009 8524 mvcc.cc:205] Tried to move back new op lower bound from 7298557839983517696 to 7298557839862775808. Current Snapshot: MvccSnapshot[applied={T|T < 7298557839983517696 or (T in {7298557839983517696})}]
I20260619 13:07:52.666265 8515 tablet_replica.cc:439] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } } }
I20260619 13:07:52.666507 8515 tablet_replica.cc:442] T 42f98b74fffc43feb0590b30e1a69af4 P fe7eb1331beb4fe788f24db3c0b9211f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:07:52.676257 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.670709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.97.3:43851)
0619 13:07:52.671227 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.671231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.671244 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:07:52.671315 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.671318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.671385 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.671556 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:52.671948 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.672709 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.674154 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.674547 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.674688 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.675079 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.675660 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.675835 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.675884 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.675959 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":284,"threads_started":1}
I20260619 13:07:52.692934 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.687025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36575 (local address 127.0.97.2:45753)
0619 13:07:52.687184 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.687193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.687786 (+ 593us) server_negotiation.cc:407] Connection header received
0619 13:07:52.687995 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.688001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.688072 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.688167 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:52.688627 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.689433 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.691002 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.691571 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.691713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.692147 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.692273 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.692373 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.692520 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.692602 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:07:52.693005 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.687062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.97.3:36575)
0619 13:07:52.687663 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.687679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.687857 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.688201 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.688205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.688217 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:52.688485 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.688493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.689570 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.689575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.690850 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.690861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.691765 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.691773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.691906 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.691960 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.692375 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.692388 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.692491 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.692624 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":332,"threads_started":1}
I20260619 13:07:52.702874 8157 raft_consensus.cc:1261] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Refusing update from remote peer fe7eb1331beb4fe788f24db3c0b9211f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.703013 8301 raft_consensus.cc:1261] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Refusing update from remote peer fe7eb1331beb4fe788f24db3c0b9211f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:52.703681 8540 consensus_queue.cc:1035] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.704036 8515 consensus_queue.cc:1035] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:52.736541 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.730103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50201 (local address 127.0.97.3:43851)
0619 13:07:52.730291 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.730299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.730792 (+ 493us) server_negotiation.cc:407] Connection header received
0619 13:07:52.731044 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.731050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.731131 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.731251 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:52.731841 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.732758 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.734367 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.734900 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.735033 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.735693 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.735828 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.735921 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.736122 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.736233 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:07:52.736620 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.730073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43851 (local address 127.0.97.2:50201)
0619 13:07:52.730698 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:52.730718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:52.730896 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:52.731272 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:52.731276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:52.731289 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:52.731645 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.731657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.732877 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.732883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.734224 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:52.734235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.735091 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.735099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:52.735334 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.735403 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:52.736000 (+ 597us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:52.736012 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 13:07:52.736100 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:07:52.736243 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":383,"threads_started":1}
I20260619 13:07:52.774298 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:52.774617 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:52.778322 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Sending DeleteTablet for 3 replicas of tablet 6daa40f3aa814c528811d20a87b3aa18
I20260619 13:07:52.779318 8137 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 13:07:52.779615 8534 tablet_replica.cc:330] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: stopping tablet replica
I20260619 13:07:52.779614 8275 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50052
I20260619 13:07:52.779807 8534 raft_consensus.cc:2229] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:52.779918 8535 tablet_replica.cc:330] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: stopping tablet replica
I20260619 13:07:52.779996 8534 raft_consensus.cc:2258] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.780133 8535 raft_consensus.cc:2229] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:52.780332 8535 raft_consensus.cc:2258] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.781283 8534 ts_tablet_manager.cc:1905] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.781869 8535 ts_tablet_manager.cc:1905] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.788676 8534 ts_tablet_manager.cc:1918] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:52.788839 8534 log.cc:1196] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/6daa40f3aa814c528811d20a87b3aa18
I20260619 13:07:52.789256 8534 ts_tablet_manager.cc:1939] T 6daa40f3aa814c528811d20a87b3aa18 P c3d5298705fb4d738223441bb2c37416: Deleting consensus metadata
I20260619 13:07:52.790462 7987 catalog_manager.cc:4841] TS c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935): tablet 6daa40f3aa814c528811d20a87b3aa18 (table test-table [id=3710ce799a3d42458854422d5f5900cf]) successfully deleted
I20260619 13:07:52.792069 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daa40f3aa814c528811d20a87b3aa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48852
I20260619 13:07:52.792418 8533 tablet_replica.cc:330] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: stopping tablet replica
I20260619 13:07:52.792649 8533 raft_consensus.cc:2229] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:52.793258 8533 raft_consensus.cc:2258] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:52.794554 8533 ts_tablet_manager.cc:1905] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:52.801505 8535 ts_tablet_manager.cc:1918] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:52.801710 8535 log.cc:1196] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/6daa40f3aa814c528811d20a87b3aa18
I20260619 13:07:52.802042 8533 ts_tablet_manager.cc:1918] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:52.802210 8533 log.cc:1196] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/6daa40f3aa814c528811d20a87b3aa18
I20260619 13:07:52.802655 8533 ts_tablet_manager.cc:1939] T 6daa40f3aa814c528811d20a87b3aa18 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting consensus metadata
I20260619 13:07:52.803798 7987 catalog_manager.cc:4841] TS fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851): tablet 6daa40f3aa814c528811d20a87b3aa18 (table test-table [id=3710ce799a3d42458854422d5f5900cf]) successfully deleted
I20260619 13:07:52.806038 8535 ts_tablet_manager.cc:1939] T 6daa40f3aa814c528811d20a87b3aa18 P 3684f3b2841e4014907151033a2c01d7: Deleting consensus metadata
I20260619 13:07:52.807282 7984 catalog_manager.cc:4841] TS 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753): tablet 6daa40f3aa814c528811d20a87b3aa18 (table test-table [id=3710ce799a3d42458854422d5f5900cf]) successfully deleted
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:07:52.866397 8095 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:07:52.853108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.0.97.62:40059)
0619 13:07:52.853308 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.853316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.853979 (+ 663us) server_negotiation.cc:407] Connection header received
0619 13:07:52.854164 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.854170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.854259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.854371 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:52.856100 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.857289 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.859252 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.859960 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.860109 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.866112 (+ 6003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42734: BlockingRecv error: recv got EOF from 127.0.0.1:42734 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:07:52 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:07:52.899623 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:52.871344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.97.62:40059)
0619 13:07:52.871529 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:52.871535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:52.874204 (+ 2669us) server_negotiation.cc:407] Connection header received
0619 13:07:52.874374 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:52.874379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:52.874463 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:52.874572 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:52.876110 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.877288 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.879298 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:52.879962 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:52.880093 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:52.885304 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:52.885313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:52.885319 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:52.885364 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:52.888901 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.889747 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.889753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.889757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.889932 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:52.890372 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:52.890377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:52.890381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:52.890792 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:52.898740 (+ 7948us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:52.898908 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 13:07:52.899222 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:07:52.904232 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:52.904965 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:52.913972 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb (DEFAULT_TABLE table=test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.914460 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d376f5e9c64a95b39e9ec469bd60eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.920640 8275 tablet_service.cc:1467] Processing CreateTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb (DEFAULT_TABLE table=test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.921092 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d376f5e9c64a95b39e9ec469bd60eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.921855 8137 tablet_service.cc:1467] Processing CreateTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb (DEFAULT_TABLE table=test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:52.922302 8137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d376f5e9c64a95b39e9ec469bd60eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:52.936493 8511 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Bootstrap starting.
I20260619 13:07:52.938216 8511 tablet_bootstrap.cc:654] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.941833 8509 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Bootstrap starting.
I20260619 13:07:52.943483 8511 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: No bootstrap required, opened a new log
I20260619 13:07:52.943599 8509 tablet_bootstrap.cc:654] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.944756 8512 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Bootstrap starting.
I20260619 13:07:52.944957 8511 ts_tablet_manager.cc:1397] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:07:52.945930 8511 raft_consensus.cc:348] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.946305 8511 raft_consensus.cc:374] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.946441 8511 raft_consensus.cc:729] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.946583 8512 tablet_bootstrap.cc:654] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:52.946913 8511 consensus_queue.cc:260] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.947652 8511 ts_tablet_manager.cc:1428] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:52.960210 8509 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: No bootstrap required, opened a new log
I20260619 13:07:52.960477 8509 ts_tablet_manager.cc:1397] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20260619 13:07:52.961444 8509 raft_consensus.cc:348] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.961815 8509 raft_consensus.cc:374] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.961962 8509 raft_consensus.cc:729] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.962333 8509 consensus_queue.cc:260] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.964196 8509 ts_tablet_manager.cc:1428] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 13:07:52.965556 8512 tablet_bootstrap.cc:492] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: No bootstrap required, opened a new log
I20260619 13:07:52.965747 8512 ts_tablet_manager.cc:1397] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20260619 13:07:52.966619 8512 raft_consensus.cc:348] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.966914 8512 raft_consensus.cc:374] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:52.967044 8512 raft_consensus.cc:729] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Initialized, Role: FOLLOWER
I20260619 13:07:52.967294 8512 consensus_queue.cc:260] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:52.967824 8512 ts_tablet_manager.cc:1428] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:07:53.339008 8517 raft_consensus.cc:482] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:53.339259 8517 raft_consensus.cc:504] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.340706 8517 leader_election.cc:288] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:53.359138 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.341099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55117 (local address 127.0.97.3:43851)
0619 13:07:53.354117 (+ 13018us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.354124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.354144 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:07:53.354219 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.354222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.354280 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.354408 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:53.354886 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.355603 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.357215 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.358168 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.358323 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.358424 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.358694 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.358828 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12335,"thread_start_us":12206,"threads_started":1}
I20260619 13:07:53.359401 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.341201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53369 (local address 127.0.97.2:45753)
0619 13:07:53.353927 (+ 12726us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.353934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.354374 (+ 440us) server_negotiation.cc:407] Connection header received
0619 13:07:53.354570 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.354575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.354650 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.354770 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:07:53.355277 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.355939 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.357546 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.358468 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.358618 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.358744 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.359015 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.359150 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12590,"thread_start_us":12731,"threads_started":1}
I20260619 13:07:53.359210 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.340993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43851 (local address 127.0.97.1:55117)
0619 13:07:53.353792 (+ 12799us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:53.353818 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:53.354018 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:53.354427 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:53.354431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:53.354457 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:53.354718 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.354726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.355751 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.355757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.357056 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.357067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.358355 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.358360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.358527 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.358550 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:07:53.358674 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 13:07:53.358822 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12664,"mutex_wait_us":2,"thread_start_us":12974,"threads_started":1}
I20260619 13:07:53.359843 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.340993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.97.1:53369)
0619 13:07:53.354244 (+ 13251us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:53.354261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:53.354433 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:53.354785 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:53.354789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:53.354809 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:07:53.355103 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.355111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.356077 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.356084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.357389 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.357399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.358681 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.358688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.358852 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.358875 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:07:53.358992 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 13:07:53.359122 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13146,"mutex_wait_us":33,"thread_start_us":12535,"threads_started":1}
I20260619 13:07:53.360133 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d376f5e9c64a95b39e9ec469bd60eb" candidate_uuid: "c3d5298705fb4d738223441bb2c37416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" is_pre_election: true
I20260619 13:07:53.360431 8435 raft_consensus.cc:2454] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d5298705fb4d738223441bb2c37416 in term 0.
I20260619 13:07:53.360421 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d376f5e9c64a95b39e9ec469bd60eb" candidate_uuid: "c3d5298705fb4d738223441bb2c37416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3684f3b2841e4014907151033a2c01d7" is_pre_election: true
I20260619 13:07:53.360671 8301 raft_consensus.cc:2454] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d5298705fb4d738223441bb2c37416 in term 0.
I20260619 13:07:53.361020 8090 leader_election.cc:302] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d5298705fb4d738223441bb2c37416, fe7eb1331beb4fe788f24db3c0b9211f; no voters:
I20260619 13:07:53.361490 8517 raft_consensus.cc:2781] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:53.361656 8517 raft_consensus.cc:482] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:53.361755 8517 raft_consensus.cc:3037] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.365679 8517 raft_consensus.cc:504] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.366362 8517 leader_election.cc:288] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [CANDIDATE]: Term 1 election: Requested vote from peers 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:53.366976 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d376f5e9c64a95b39e9ec469bd60eb" candidate_uuid: "c3d5298705fb4d738223441bb2c37416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
I20260619 13:07:53.366993 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d376f5e9c64a95b39e9ec469bd60eb" candidate_uuid: "c3d5298705fb4d738223441bb2c37416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3684f3b2841e4014907151033a2c01d7"
I20260619 13:07:53.367203 8435 raft_consensus.cc:3037] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.367203 8301 raft_consensus.cc:3037] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.370170 8435 raft_consensus.cc:2454] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d5298705fb4d738223441bb2c37416 in term 1.
I20260619 13:07:53.370170 8301 raft_consensus.cc:2454] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d5298705fb4d738223441bb2c37416 in term 1.
I20260619 13:07:53.370661 8090 leader_election.cc:302] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416, fe7eb1331beb4fe788f24db3c0b9211f; no voters:
I20260619 13:07:53.370990 8517 raft_consensus.cc:2781] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:53.371408 8517 raft_consensus.cc:686] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 1 LEADER]: Becoming Leader. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Running, Role: LEADER
I20260619 13:07:53.371788 8517 consensus_queue.cc:237] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.375685 8000 catalog_manager.cc:5495] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d5298705fb4d738223441bb2c37416 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c3d5298705fb4d738223441bb2c37416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:53.490680 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.484738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.97.1:33935)
0619 13:07:53.485242 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.485247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.485268 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:07:53.485368 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.485373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.485475 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.485752 (+ 277us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:53.486140 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.486942 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.488278 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.488700 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.488802 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.489209 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:53.489921 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:53.490038 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.490193 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.490302 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":284,"threads_started":1}
I20260619 13:07:53.494771 8435 raft_consensus.cc:1261] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Refusing update from remote peer c3d5298705fb4d738223441bb2c37416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:53.494935 8301 raft_consensus.cc:1261] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Refusing update from remote peer c3d5298705fb4d738223441bb2c37416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:53.495607 8587 consensus_queue.cc:1035] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:53.496060 8517 consensus_queue.cc:1035] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:53.520018 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:53.520330 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:53.523756 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Sending DeleteTablet for 3 replicas of tablet 03d376f5e9c64a95b39e9ec469bd60eb
I20260619 13:07:53.524526 8137 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 13:07:53.524720 8275 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50052
I20260619 13:07:53.524731 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d376f5e9c64a95b39e9ec469bd60eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48852
I20260619 13:07:53.527940 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:07:53.528755 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:07:53.537742 8276 tablet_service.cc:1467] Processing CreateTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 (DEFAULT_TABLE table=test-table [id=542c67f458f547e495ec7e108c5573a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:53.537881 8136 tablet_service.cc:1467] Processing CreateTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 (DEFAULT_TABLE table=test-table [id=542c67f458f547e495ec7e108c5573a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:53.538419 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc6648d690c4de0afd66d2bac48a9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:53.538422 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc6648d690c4de0afd66d2bac48a9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:53.542395 8414 tablet_service.cc:1467] Processing CreateTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 (DEFAULT_TABLE table=test-table [id=542c67f458f547e495ec7e108c5573a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:07:53.542840 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc6648d690c4de0afd66d2bac48a9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:53.553063 8606 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Bootstrap starting.
I20260619 13:07:53.553989 8604 tablet_replica.cc:330] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: stopping tablet replica
I20260619 13:07:53.554277 8604 raft_consensus.cc:2229] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:53.556600 8606 tablet_bootstrap.cc:654] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:53.556975 8604 raft_consensus.cc:2258] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.557626 8605 tablet_replica.cc:330] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: stopping tablet replica
I20260619 13:07:53.558336 8604 ts_tablet_manager.cc:1905] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.561105 8608 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Bootstrap starting.
I20260619 13:07:53.561865 8605 raft_consensus.cc:2229] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:53.562255 8605 raft_consensus.cc:2258] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.563700 8608 tablet_bootstrap.cc:654] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:53.564234 8605 ts_tablet_manager.cc:1905] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.565016 8603 tablet_replica.cc:330] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: stopping tablet replica
I20260619 13:07:53.565236 8603 raft_consensus.cc:2229] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:53.565868 8603 raft_consensus.cc:2258] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.566967 8607 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Bootstrap starting.
I20260619 13:07:53.569937 8603 ts_tablet_manager.cc:1905] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.572732 8606 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: No bootstrap required, opened a new log
I20260619 13:07:53.572882 8606 ts_tablet_manager.cc:1397] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260619 13:07:53.573817 8606 raft_consensus.cc:348] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.574107 8606 raft_consensus.cc:374] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:53.574227 8606 raft_consensus.cc:729] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5298705fb4d738223441bb2c37416, State: Initialized, Role: FOLLOWER
I20260619 13:07:53.574486 8606 consensus_queue.cc:260] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.575280 8606 ts_tablet_manager.cc:1428] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 13:07:53.576148 8604 ts_tablet_manager.cc:1918] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:53.576300 8604 log.cc:1196] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/03d376f5e9c64a95b39e9ec469bd60eb
I20260619 13:07:53.576829 8604 ts_tablet_manager.cc:1939] T 03d376f5e9c64a95b39e9ec469bd60eb P fe7eb1331beb4fe788f24db3c0b9211f: Deleting consensus metadata
I20260619 13:07:53.577085 8607 tablet_bootstrap.cc:654] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:53.578157 7987 catalog_manager.cc:4841] TS fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851): tablet 03d376f5e9c64a95b39e9ec469bd60eb (table test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]) successfully deleted
I20260619 13:07:53.579550 8608 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: No bootstrap required, opened a new log
I20260619 13:07:53.579783 8608 ts_tablet_manager.cc:1397] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20260619 13:07:53.580441 8607 tablet_bootstrap.cc:492] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: No bootstrap required, opened a new log
I20260619 13:07:53.580595 8607 ts_tablet_manager.cc:1397] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20260619 13:07:53.580710 8603 ts_tablet_manager.cc:1918] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:53.580847 8603 log.cc:1196] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/03d376f5e9c64a95b39e9ec469bd60eb
I20260619 13:07:53.581328 8603 ts_tablet_manager.cc:1939] T 03d376f5e9c64a95b39e9ec469bd60eb P c3d5298705fb4d738223441bb2c37416: Deleting consensus metadata
I20260619 13:07:53.581435 8607 raft_consensus.cc:348] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.581713 8607 raft_consensus.cc:374] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:53.581859 8607 raft_consensus.cc:729] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7eb1331beb4fe788f24db3c0b9211f, State: Initialized, Role: FOLLOWER
I20260619 13:07:53.582062 8608 raft_consensus.cc:348] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.582156 8607 consensus_queue.cc:260] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.582708 8607 ts_tablet_manager.cc:1428] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:07:53.582926 8608 raft_consensus.cc:374] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:53.583036 8608 raft_consensus.cc:729] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Initialized, Role: FOLLOWER
I20260619 13:07:53.583282 8608 consensus_queue.cc:260] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.583817 8608 ts_tablet_manager.cc:1428] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 13:07:53.584329 7987 catalog_manager.cc:4841] TS c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935): tablet 03d376f5e9c64a95b39e9ec469bd60eb (table test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]) successfully deleted
I20260619 13:07:53.586926 8605 ts_tablet_manager.cc:1918] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:07:53.587080 8605 log.cc:1196] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/03d376f5e9c64a95b39e9ec469bd60eb
I20260619 13:07:53.587566 8605 ts_tablet_manager.cc:1939] T 03d376f5e9c64a95b39e9ec469bd60eb P 3684f3b2841e4014907151033a2c01d7: Deleting consensus metadata
I20260619 13:07:53.588780 7984 catalog_manager.cc:4841] TS 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753): tablet 03d376f5e9c64a95b39e9ec469bd60eb (table test-table [id=d1ed4bcc89c648e8ac1321ac6c05d10a]) successfully deleted
I20260619 13:07:53.674324 8520 raft_consensus.cc:482] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:07:53.674592 8520 raft_consensus.cc:504] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.675412 8520 leader_election.cc:288] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:53.676035 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc6648d690c4de0afd66d2bac48a9b4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" is_pre_election: true
I20260619 13:07:53.676017 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc6648d690c4de0afd66d2bac48a9b4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416" is_pre_election: true
I20260619 13:07:53.676299 8435 raft_consensus.cc:2454] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:53.676347 8156 raft_consensus.cc:2454] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 0.
I20260619 13:07:53.676755 8230 leader_election.cc:302] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, fe7eb1331beb4fe788f24db3c0b9211f; no voters:
I20260619 13:07:53.677094 8520 raft_consensus.cc:2781] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:07:53.677214 8520 raft_consensus.cc:482] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:07:53.677335 8520 raft_consensus.cc:3037] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.680359 8520 raft_consensus.cc:504] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.681020 8520 leader_election.cc:288] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Requested vote from peers c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935), fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851)
I20260619 13:07:53.681679 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc6648d690c4de0afd66d2bac48a9b4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7eb1331beb4fe788f24db3c0b9211f"
I20260619 13:07:53.681677 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc6648d690c4de0afd66d2bac48a9b4" candidate_uuid: "3684f3b2841e4014907151033a2c01d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d5298705fb4d738223441bb2c37416"
I20260619 13:07:53.681902 8435 raft_consensus.cc:3037] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.681916 8156 raft_consensus.cc:3037] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:53.685006 8435 raft_consensus.cc:2454] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:53.685127 8156 raft_consensus.cc:2454] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3684f3b2841e4014907151033a2c01d7 in term 1.
I20260619 13:07:53.685526 8230 leader_election.cc:302] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3684f3b2841e4014907151033a2c01d7, c3d5298705fb4d738223441bb2c37416; no voters:
I20260619 13:07:53.685817 8520 raft_consensus.cc:2781] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:53.686312 8520 raft_consensus.cc:686] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3684f3b2841e4014907151033a2c01d7, State: Running, Role: LEADER
I20260619 13:07:53.686797 8520 consensus_queue.cc:237] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } }
I20260619 13:07:53.689729 8000 catalog_manager.cc:5495] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3684f3b2841e4014907151033a2c01d7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3684f3b2841e4014907151033a2c01d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3684f3b2841e4014907151033a2c01d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260619 13:07:53.725956 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.720063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.0.97.2:45753)
0619 13:07:53.720533 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.720538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.720551 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:07:53.720594 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.720598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.720652 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.720718 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:53.721157 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.721978 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.723977 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.724477 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.724656 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.725106 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:53.725211 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:53.725285 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.725529 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.725655 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260619 13:07:53.740622 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.734743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44847 (local address 127.0.97.2:45753)
0619 13:07:53.734947 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.734957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.735397 (+ 440us) server_negotiation.cc:407] Connection header received
0619 13:07:53.735598 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.735601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.735664 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.735738 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:07:53.736264 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.736968 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.738890 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.739303 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.739434 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.739931 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:07:53.740047 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:07:53.740127 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.740287 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.740350 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":33}
I20260619 13:07:53.740861 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.734528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45753 (local address 127.0.97.2:44847)
0619 13:07:53.735293 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:53.735314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:53.735454 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:53.735785 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:53.735790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:53.735803 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:07:53.736132 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.736143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.737103 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.737110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.738734 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:53.738748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.739504 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.739511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:53.739698 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.739748 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:07:53.740159 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:07:53.740169 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:07:53.740254 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:07:53.740392 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":3,"thread_start_us":436,"threads_started":1}
I20260619 13:07:53.749352 8435 raft_consensus.cc:1261] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:53.749451 8156 raft_consensus.cc:1261] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Refusing update from remote peer 3684f3b2841e4014907151033a2c01d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:07:53.750000 8614 consensus_queue.cc:1035] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7eb1331beb4fe788f24db3c0b9211f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:53.750347 8520 consensus_queue.cc:1035] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d5298705fb4d738223441bb2c37416" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:07:53.811527 8000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:53.811844 8000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:07:53.815135 8000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f8d661028c44f09540c96d68f8f52f: Sending DeleteTablet for 3 replicas of tablet ffc6648d690c4de0afd66d2bac48a9b4
I20260619 13:07:53.816000 8137 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 13:07:53.816021 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50052
I20260619 13:07:53.816517 8603 tablet_replica.cc:330] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: stopping tablet replica
I20260619 13:07:53.816619 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc6648d690c4de0afd66d2bac48a9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:07:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48852
I20260619 13:07:53.816742 8603 raft_consensus.cc:2229] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:53.816941 8603 raft_consensus.cc:2258] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.817072 8604 tablet_replica.cc:330] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: stopping tablet replica
I20260619 13:07:53.817289 8604 raft_consensus.cc:2229] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:07:53.817472 8604 raft_consensus.cc:2258] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.817752 8605 tablet_replica.cc:330] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: stopping tablet replica
I20260619 13:07:53.817909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8071
I20260619 13:07:53.817941 8605 raft_consensus.cc:2229] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:07:53.818225 8603 ts_tablet_manager.cc:1905] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.818249 8605 raft_consensus.cc:2258] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:07:53.821713 8605 ts_tablet_manager.cc:1905] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.823535 8604 ts_tablet_manager.cc:1905] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:07:53.825518 8603 ts_tablet_manager.cc:1918] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:53.825691 8603 log.cc:1196] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/ffc6648d690c4de0afd66d2bac48a9b4
I20260619 13:07:53.826133 8603 ts_tablet_manager.cc:1939] T ffc6648d690c4de0afd66d2bac48a9b4 P c3d5298705fb4d738223441bb2c37416: Deleting consensus metadata
I20260619 13:07:53.827134 7987 catalog_manager.cc:4841] TS c3d5298705fb4d738223441bb2c37416 (127.0.97.1:33935): tablet ffc6648d690c4de0afd66d2bac48a9b4 (table test-table [id=542c67f458f547e495ec7e108c5573a6]) successfully deleted
I20260619 13:07:53.828338 8605 ts_tablet_manager.cc:1918] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:53.828477 8605 log.cc:1196] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/ffc6648d690c4de0afd66d2bac48a9b4
I20260619 13:07:53.828881 8605 ts_tablet_manager.cc:1939] T ffc6648d690c4de0afd66d2bac48a9b4 P 3684f3b2841e4014907151033a2c01d7: Deleting consensus metadata
I20260619 13:07:53.829909 7984 catalog_manager.cc:4841] TS 3684f3b2841e4014907151033a2c01d7 (127.0.97.2:45753): tablet ffc6648d690c4de0afd66d2bac48a9b4 (table test-table [id=542c67f458f547e495ec7e108c5573a6]) successfully deleted
I20260619 13:07:53.830910 8604 ts_tablet_manager.cc:1918] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:07:53.831115 8604 log.cc:1196] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/ffc6648d690c4de0afd66d2bac48a9b4
Jun 19 13:07:53 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 13:07:53.831540 8604 ts_tablet_manager.cc:1939] T ffc6648d690c4de0afd66d2bac48a9b4 P fe7eb1331beb4fe788f24db3c0b9211f: Deleting consensus metadata
I20260619 13:07:53.832713 7987 catalog_manager.cc:4841] TS fe7eb1331beb4fe788f24db3c0b9211f (127.0.97.3:43851): tablet ffc6648d690c4de0afd66d2bac48a9b4 (table test-table [id=542c67f458f547e495ec7e108c5573a6]) successfully deleted
I20260619 13:07:53.839677 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.819832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.0.97.1:33935)
0619 13:07:53.820020 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.820026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.820049 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:07:53.820109 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.820114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.820208 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.820326 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:53.822363 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.823523 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.825632 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.826390 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.826500 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:53.832782 (+ 6282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:53.832798 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:53.832829 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:53.832882 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:53.836530 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:53.837432 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:53.837441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:53.837443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:53.837686 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:53.838105 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:53.838108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:53.838110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:53.838529 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:53.838804 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:53.839140 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 13:07:53.839334 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:07:53.840381 8202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:53.970423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8071
I20260619 13:07:53.993650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8211
Jun 19 13:07:54 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 13:07:54.008829 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:53.994470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.0.97.2:45753)
0619 13:07:53.994658 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:53.994664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:53.994707 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:07:53.994927 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:53.994933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:53.995014 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:53.995126 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:53.996392 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.997100 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.998425 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:53.998817 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:53.998955 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:54.003109 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:54.003124 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:54.003141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:54.003177 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:54.006142 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:54.006965 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:54.006973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:54.006975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:54.007121 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:54.007479 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:54.007482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:54.007484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:54.007877 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:54.008130 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:54.008379 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 13:07:54.008581 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:07:54.009588 8341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:54.160641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8211
I20260619 13:07:54.186764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8350
Jun 19 13:07:54 dist-test-slave-q11v krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 13:07:54.202631 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:54.187558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.97.3:43851)
0619 13:07:54.188025 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:54.188030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:54.188045 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:07:54.188100 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:54.188104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:54.188165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:54.188266 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:54.189679 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:54.190490 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:54.191935 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:54.192346 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:54.192450 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:54.196950 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:54.196965 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:54.196981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:54.197019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:54.199865 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:54.200527 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:54.200534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:54.200537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:54.200678 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:54.201006 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:54.201009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:54.201011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:54.201340 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:54.201556 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:54.202038 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 13:07:54.202219 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260619 13:07:54.203369 8480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:54.326381 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8350
I20260619 13:07:54.348585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 7968
I20260619 13:07:54.349857 8040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:54.466308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 7968
2026-06-19T13:07:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5962 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:54 dist-test-slave-q11v krb5kdc[8631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:54 dist-test-slave-q11v krb5kdc[8631](info): set up 2 sockets
Jun 19 13:07:54 dist-test-slave-q11v krb5kdc[8631](info): commencing operation
krb5kdc: starting...
W20260619 13:07:55.574599 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:55 dist-test-slave-q11v krb5kdc[8631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:55.797251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44409
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:56.125839 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:56.126228 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:56.126405 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:56.133787 8644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:56.133958 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:56.134071 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:56.134173 8644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:56.134267 8644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:56.147373 8644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:56.149053 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:56.150975 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:56.163353 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:56.163491 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:56.163936 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:56.164723 8644 server_base.cc:1034] running on GCE node
I20260619 13:07:56.165541 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:56.167590 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:56.168844 8644 hybrid_clock.cc:648] HybridClock initialized: now 1781874476168778 us; error 58 us; skew 500 ppm
I20260619 13:07:56.169165 8644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T13:07:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1767 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:07:56 dist-test-slave-q11v krb5kdc[8658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:07:56 dist-test-slave-q11v krb5kdc[8658](info): set up 2 sockets
Jun 19 13:07:56 dist-test-slave-q11v krb5kdc[8658](info): commencing operation
krb5kdc: starting...
W20260619 13:07:57.346609 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:07:57 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:07:57.569787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:57.915547 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:57.915848 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:57.915937 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:57.923213 8671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:07:57.923331 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:57.923377 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:57.923434 8671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:07:57.923478 8671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:07:57.933154 8671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:57.934594 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:57.935935 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:57.947535 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:57.947618 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:57.948138 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:57.948495 8671 server_base.cc:1034] running on GCE node
I20260619 13:07:57.949297 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:57.950892 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:57.952126 8671 hybrid_clock.cc:648] HybridClock initialized: now 1781874477952049 us; error 57 us; skew 500 ppm
I20260619 13:07:57.952582 8671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:57 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:57.955998 8671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:07:57.957576 8671 webserver.cc:460] Webserver started at http://127.0.97.62:45745/ using document root <none> and password file <none>
I20260619 13:07:57.958370 8671 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:57.958474 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:57.958756 8671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:57.961961 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "4cd4beb82db046a28b0403c14417a151"
format_stamp: "Formatted at 2026-06-19 13:07:57 on dist-test-slave-q11v"
server_key: "e8947d598fe1d7fc4bc59f9f9d8df7ba"
server_key_iv: "4823f2388f9238e5024979a178fdb66b"
server_key_version: "encryptionkey@0"
I20260619 13:07:57.962735 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "4cd4beb82db046a28b0403c14417a151"
format_stamp: "Formatted at 2026-06-19 13:07:57 on dist-test-slave-q11v"
server_key: "e8947d598fe1d7fc4bc59f9f9d8df7ba"
server_key_iv: "4823f2388f9238e5024979a178fdb66b"
server_key_version: "encryptionkey@0"
I20260619 13:07:57.968030 8671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 13:07:57.971617 8686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:57.973068 8671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:07:57.973306 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "4cd4beb82db046a28b0403c14417a151"
format_stamp: "Formatted at 2026-06-19 13:07:57 on dist-test-slave-q11v"
server_key: "e8947d598fe1d7fc4bc59f9f9d8df7ba"
server_key_iv: "4823f2388f9238e5024979a178fdb66b"
server_key_version: "encryptionkey@0"
I20260619 13:07:57.973519 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:57.998019 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:58.002506 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:58.002811 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:58.022159 8671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34673
I20260619 13:07:58.022215 8737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34673 every 8 connection(s)
I20260619 13:07:58.023595 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:07:58.027753 8738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:07:58.031625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8671
I20260619 13:07:58.031874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:07:58.032510 388 external_mini_cluster.cc:1428] Setting key c2be5773a5cbfdd661efb5b5b7a7dd90
I20260619 13:07:58.042371 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Bootstrap starting.
I20260619 13:07:58.046494 8738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Neither blocks nor log segments found. Creating new log.
I20260619 13:07:58.047616 8738 log.cc:824] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:07:58 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:58.051399 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: No bootstrap required, opened a new log
I20260619 13:07:58.058982 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:58.034756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.97.62:34673)
0619 13:07:58.035592 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:58.035609 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:58.035669 (+ 60us) server_negotiation.cc:407] Connection header received
0619 13:07:58.037722 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:58.037764 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:58.038415 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:58.038944 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:58.040347 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.042473 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.043799 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.044975 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.045186 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:58.049233 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:58.049244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:58.049265 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:58.049315 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:58.053573 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:58.054336 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:58.054345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:58.054356 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:58.054597 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:58.055093 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:58.055099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:58.055103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:58.055922 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:58.056269 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:58.056856 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 13:07:58.057388 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":349,"threads_started":1}
I20260619 13:07:58.062855 8738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:07:58.063256 8738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:07:58.063369 8738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4beb82db046a28b0403c14417a151, State: Initialized, Role: FOLLOWER
I20260619 13:07:58.064023 8738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:07:58.064249 8738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:07:58.064410 8738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:07:58.064615 8738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:07:58.066988 8738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:07:58.067524 8738 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4beb82db046a28b0403c14417a151; no voters:
I20260619 13:07:58.068264 8738 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:07:58.068399 8743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:07:58.069427 8743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd4beb82db046a28b0403c14417a151, State: Running, Role: LEADER
I20260619 13:07:58.070117 8743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:07:58.070839 8738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:07:58.073345 8745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd4beb82db046a28b0403c14417a151. Latest consensus state: current_term: 1 leader_uuid: "4cd4beb82db046a28b0403c14417a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260619 13:07:58.073405 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd4beb82db046a28b0403c14417a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260619 13:07:58.073787 8745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:58.073848 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: This master's current role is: LEADER
I20260619 13:07:58.080292 8749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:07:58.086071 8749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:07:58.095273 8749 catalog_manager.cc:1347] Generated new cluster ID: 53546d7a2aad4ef2a1bf2fc6223fb69a
I20260619 13:07:58.095407 8749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:07:58.107801 8749 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:07:58.109300 8749 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:07:58.119328 8749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Generated new TSK 0
I20260619 13:07:58.120227 8749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:07:58.175297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:58.504988 8764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:58.505281 8764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:58.505446 8764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:58.512848 8764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:58.512960 8764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:58.513168 8764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:07:58.522889 8764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:58.524204 8764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:58.525532 8764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:58.538637 8769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:58.538672 8770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:58.538619 8772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:58.539431 8764 server_base.cc:1034] running on GCE node
I20260619 13:07:58.540122 8764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:58.541078 8764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:58.542276 8764 hybrid_clock.cc:648] HybridClock initialized: now 1781874478542231 us; error 36 us; skew 500 ppm
I20260619 13:07:58.542590 8764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:58 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:58.545852 8764 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:07:58.547286 8764 webserver.cc:460] Webserver started at http://127.0.97.1:36169/ using document root <none> and password file <none>
I20260619 13:07:58.547933 8764 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:58.548031 8764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:58.548287 8764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:58.551179 8764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "265dd66b54d748c98e1f95c88fc8c830"
format_stamp: "Formatted at 2026-06-19 13:07:58 on dist-test-slave-q11v"
server_key: "8710748635d368732ec2238a8d697ac4"
server_key_iv: "ea75b5246de639f53722dc6b23ca697c"
server_key_version: "encryptionkey@0"
I20260619 13:07:58.551899 8764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "265dd66b54d748c98e1f95c88fc8c830"
format_stamp: "Formatted at 2026-06-19 13:07:58 on dist-test-slave-q11v"
server_key: "8710748635d368732ec2238a8d697ac4"
server_key_iv: "ea75b5246de639f53722dc6b23ca697c"
server_key_version: "encryptionkey@0"
I20260619 13:07:58.557045 8764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:58.561151 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:58.562584 8764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:58.562790 8764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "265dd66b54d748c98e1f95c88fc8c830"
format_stamp: "Formatted at 2026-06-19 13:07:58 on dist-test-slave-q11v"
server_key: "8710748635d368732ec2238a8d697ac4"
server_key_iv: "ea75b5246de639f53722dc6b23ca697c"
server_key_version: "encryptionkey@0"
I20260619 13:07:58.563000 8764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:58.574455 8764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:58.579082 8764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:58.579399 8764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:58.580415 8764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:58.581879 8764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:58.581952 8764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:58.582119 8764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:58.582180 8764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:58.616779 8764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33485
I20260619 13:07:58.616816 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33485 every 8 connection(s)
I20260619 13:07:58.618227 8764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:07:58.628369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8764
I20260619 13:07:58.628614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:07:58.629326 388 external_mini_cluster.cc:1428] Setting key ad3a5eac1ff9425904e809a0a74350ee
Jun 19 13:07:58 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:58.638924 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:58.621046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53707 (local address 127.0.97.62:34673)
0619 13:07:58.621533 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:58.621545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:58.622692 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 13:07:58.624389 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:58.624394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:58.624469 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:58.624587 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:58.626167 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.626958 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.628661 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.629122 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.629262 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:58.633175 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:58.633180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:58.633183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:58.633215 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:58.635795 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:58.636556 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:58.636566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:58.636568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:58.636685 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:58.637137 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:58.637141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:58.637144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:58.637409 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:58.637566 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:58.638336 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 13:07:58.638560 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":270,"threads_started":1}
I20260619 13:07:58.639971 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:58.621807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:53707)
0619 13:07:58.622514 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:58.622552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:58.624130 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:58.624799 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:58.624807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:58.625316 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:58.625990 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:58.626008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.627131 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.627135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:58.628442 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:58.628455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:58.629377 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:58.629385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:58.629695 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:58.630474 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:58.630499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:58.632921 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:58.635946 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:58.635956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:58.635973 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:58.636421 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:58.636797 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:58.636802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:58.636806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:58.637006 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:58.637539 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:58.637547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:58.637953 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 13:07:58.638241 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 13:07:58.638814 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":273,"threads_started":1}
I20260619 13:07:58.641711 8892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260619 13:07:58.642035 8892 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:58.642750 8892 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260619 13:07:58.645153 8703 ts_manager.cc:194] Registered new tserver with Master: 265dd66b54d748c98e1f95c88fc8c830 (127.0.97.1:33485)
I20260619 13:07:58.647643 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53707
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:07:58.681143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:59.007045 8898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:59.007351 8898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:59.007540 8898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:59.015038 8898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:59.015163 8898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:59.015353 8898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:07:59.025483 8898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:59.026816 8898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:59.028134 8898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:59.041947 8903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:59.041972 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:59.042074 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:59.043211 8898 server_base.cc:1034] running on GCE node
I20260619 13:07:59.043844 8898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:59.045044 8898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:59.046294 8898 hybrid_clock.cc:648] HybridClock initialized: now 1781874479046235 us; error 48 us; skew 500 ppm
I20260619 13:07:59.046712 8898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:59.050169 8898 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:07:59.051730 8898 webserver.cc:460] Webserver started at http://127.0.97.2:32815/ using document root <none> and password file <none>
I20260619 13:07:59.052457 8898 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:59.052608 8898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:59.052927 8898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:59.056073 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "838b8bb5032b4192a2120b49648765d4"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "401161c89978d39c975637ae4e6b2830"
server_key_iv: "7a99cc54ccb4ebb7811a867b95436e7e"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.056877 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "838b8bb5032b4192a2120b49648765d4"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "401161c89978d39c975637ae4e6b2830"
server_key_iv: "7a99cc54ccb4ebb7811a867b95436e7e"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.063030 8898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 13:07:59.066628 8913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.068260 8898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:07:59.068451 8898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "838b8bb5032b4192a2120b49648765d4"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "401161c89978d39c975637ae4e6b2830"
server_key_iv: "7a99cc54ccb4ebb7811a867b95436e7e"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.068676 8898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:59.081446 8898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:59.085579 8898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:59.085958 8898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:59.086921 8898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:59.088285 8898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:59.088397 8898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.088550 8898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:59.088625 8898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.122473 8898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42463
I20260619 13:07:59.122565 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42463 every 8 connection(s)
I20260619 13:07:59.123814 8898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:07:59.133206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 8898
I20260619 13:07:59.133389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:07:59.133930 388 external_mini_cluster.cc:1428] Setting key 6a3b4be2b352f9b6bd7c1d846441021a
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:59.146498 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.126529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42535 (local address 127.0.97.62:34673)
0619 13:07:59.126735 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:59.126740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:59.127912 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 13:07:59.129822 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:59.129829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:59.129922 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:59.130039 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:59.131533 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.132242 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.133972 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.134487 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.134645 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.138958 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:59.138969 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:59.138977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:59.139029 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:59.142884 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.143989 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.143994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.143997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.144164 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.144732 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.144736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.144738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.145020 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:59.145141 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:59.146076 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 13:07:59.146254 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:07:59.148118 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.127047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:42535)
0619 13:07:59.127739 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:59.127776 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:59.129473 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:59.130283 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:59.130291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:59.130793 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:59.131355 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:59.131371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.132394 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.132399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:59.133826 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:59.133838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.134806 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.134815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:59.135132 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.135916 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:59.135961 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:59.138696 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:59.143161 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:59.143168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:59.143182 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:59.143799 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:59.144310 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:59.144317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:59.144322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:59.144594 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:59.145158 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:59.145169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:59.145671 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 13:07:59.145975 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 13:07:59.146564 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260619 13:07:59.149695 9026 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260619 13:07:59.150015 9026 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:59.150732 9026 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260619 13:07:59.152190 8703 ts_manager.cc:194] Registered new tserver with Master: 838b8bb5032b4192a2120b49648765d4 (127.0.97.2:42463)
I20260619 13:07:59.153091 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42535
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:07:59.190201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:07:59.522575 9031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:07:59.522940 9031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:07:59.523118 9031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:07:59.530336 9031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:07:59.530498 9031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:07:59.530682 9031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:07:59.540377 9031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:07:59.541709 9031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:07:59.542987 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:07:59.556103 9039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:59.556551 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:07:59.556613 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:07:59.557327 9031 server_base.cc:1034] running on GCE node
I20260619 13:07:59.558045 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:07:59.559000 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:07:59.560247 9031 hybrid_clock.cc:648] HybridClock initialized: now 1781874479560198 us; error 47 us; skew 500 ppm
I20260619 13:07:59.560611 9031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:07:59.563889 9031 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:07:59.565433 9031 webserver.cc:460] Webserver started at http://127.0.97.3:37135/ using document root <none> and password file <none>
I20260619 13:07:59.566310 9031 fs_manager.cc:359] Metadata directory not provided
I20260619 13:07:59.566459 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:07:59.566784 9031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:07:59.569772 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "4568e987c34c4fb3908c6723decac591"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "b7216a844e77482245a9ef14ed309290"
server_key_iv: "9cdd1fc7af3272dbd4846ffbef6aba06"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.570547 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4568e987c34c4fb3908c6723decac591"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "b7216a844e77482245a9ef14ed309290"
server_key_iv: "9cdd1fc7af3272dbd4846ffbef6aba06"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.576419 9031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:07:59.580279 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.581842 9031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:07:59.582038 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "4568e987c34c4fb3908c6723decac591"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "b7216a844e77482245a9ef14ed309290"
server_key_iv: "9cdd1fc7af3272dbd4846ffbef6aba06"
server_key_version: "encryptionkey@0"
I20260619 13:07:59.582273 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:07:59.602713 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:07:59.607065 9031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:07:59.607367 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:07:59.608362 9031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:07:59.609783 9031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:07:59.609891 9031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.610049 9031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:07:59.610124 9031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:07:59.644059 9031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45147
I20260619 13:07:59.644155 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45147 every 8 connection(s)
I20260619 13:07:59.645365 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:07:59.651757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9031
I20260619 13:07:59.651947 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:07:59.652272 8892 heartbeater.cc:503] Master 127.0.97.62:34673 was elected leader, sending a full tablet report...
I20260619 13:07:59.652549 388 external_mini_cluster.cc:1428] Setting key 9d0b40ae645d62086f83c53ec71ab8ba
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:07:59.666177 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.648321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44257 (local address 127.0.97.62:34673)
0619 13:07:59.648750 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:59.648756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:59.649921 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 13:07:59.651806 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:59.651812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:59.651903 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:59.652064 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:59.654262 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.655094 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.656669 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.657080 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.657222 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.660731 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:59.660736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:59.660739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:59.660770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:59.663225 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.663950 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.663955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.663957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.664079 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.664469 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.664472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.664474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.664742 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:59.664857 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:59.665530 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 13:07:59.665782 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260619 13:07:59.667282 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.648961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:44257)
0619 13:07:59.649750 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 13:07:59.649790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:07:59.651514 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:07:59.652338 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:07:59.652351 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:07:59.653218 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:07:59.654076 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:59.654114 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.655256 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.655261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:59.656515 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:07:59.656529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.657313 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.657320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:07:59.657550 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.658271 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:07:59.658297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:07:59.660498 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:07:59.663354 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:59.663360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:59.663375 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:59.663808 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:59.664178 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:07:59.664181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:07:59.664184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:07:59.664350 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:07:59.664852 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:07:59.664857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:07:59.665164 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 13:07:59.665391 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 13:07:59.665873 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":250,"threads_started":1}
I20260619 13:07:59.668692 9159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260619 13:07:59.668956 9159 heartbeater.cc:464] Registering TS with master...
I20260619 13:07:59.669730 9159 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260619 13:07:59.671169 8703 ts_manager.cc:194] Registered new tserver with Master: 4568e987c34c4fb3908c6723decac591 (127.0.97.3:45147)
I20260619 13:07:59.672134 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44257
I20260619 13:07:59.680655 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:07:59.810389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8764
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:07:59.826505 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.811512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.97.1:33485)
0619 13:07:59.812183 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:59.812193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:59.812228 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:07:59.812417 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:59.812425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:59.812650 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:59.812822 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:59.814202 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.815231 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.816476 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.817045 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.817183 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.821188 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:59.821198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:59.821211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:59.821242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:59.823902 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.824569 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.824576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.824578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.824705 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.825061 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.825064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.825066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.825378 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:59.825659 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:59.826005 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 13:07:59.826216 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
I20260619 13:07:59.827864 8887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:07:59.911239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8764
I20260619 13:07:59.927390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8898
Jun 19 13:07:59 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:07:59.943928 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:07:59.928312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.0.97.2:42463)
0619 13:07:59.928822 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 13:07:59.928829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:07:59.928853 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:07:59.928911 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:07:59.928917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:07:59.929131 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:07:59.929302 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:07:59.930572 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.931576 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.932817 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:07:59.933353 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:07:59.933494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:07:59.938108 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:07:59.938120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:07:59.938134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:07:59.938169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:07:59.940884 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.941632 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.941644 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.941650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.941797 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:07:59.942214 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:07:59.942221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:07:59.942225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:07:59.942581 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:07:59.942916 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 13:07:59.943270 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 13:07:59.943545 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260619 13:07:59.945231 9021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:00.027200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8898
I20260619 13:08:00.042826 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9031
Jun 19 13:08:00 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:08:00.059473 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:00.043951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.97.3:45147)
0619 13:08:00.044605 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:00.044612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:00.044640 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:08:00.044707 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:00.044713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:00.044916 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:00.045059 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:00.046356 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:00.047388 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:00.048666 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:00.049243 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:00.049351 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:00.053737 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:00.053749 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:00.053764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:00.053804 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:00.056663 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:00.057272 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:00.057279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:00.057281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:00.057403 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:00.057908 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:00.057913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:00.057916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:00.058325 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:00.058590 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:00.058890 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 13:08:00.059160 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":352,"threads_started":1}
I20260619 13:08:00.060781 9154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:00.146899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9031
I20260619 13:08:00.163069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 8671
I20260619 13:08:00.164247 8733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:00.265996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 8671
I20260619 13:08:00.286692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--webserver_interface=127.0.97.62
--webserver_port=45745
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:08:00.606272 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:00.606572 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:00.606783 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:00.614122 9177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:08:00.614233 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:00.614356 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:00.614470 9177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:08:00.614589 9177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:08:00.624781 9177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:00.626226 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:00.627686 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:00.640347 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:00.640347 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:00.640692 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:00.641254 9177 server_base.cc:1034] running on GCE node
I20260619 13:08:00.642191 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:00.643643 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:00.644892 9177 hybrid_clock.cc:648] HybridClock initialized: now 1781874480644846 us; error 65 us; skew 500 ppm
I20260619 13:08:00.645258 9177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:00 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:00.648412 9177 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:08:00.649859 9177 webserver.cc:460] Webserver started at http://127.0.97.62:45745/ using document root <none> and password file <none>
I20260619 13:08:00.650506 9177 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:00.650617 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:00.655911 9177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 13:08:00.658671 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:00.660105 9177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:08:00.660241 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "4cd4beb82db046a28b0403c14417a151"
format_stamp: "Formatted at 2026-06-19 13:07:57 on dist-test-slave-q11v"
server_key: "e8947d598fe1d7fc4bc59f9f9d8df7ba"
server_key_iv: "4823f2388f9238e5024979a178fdb66b"
server_key_version: "encryptionkey@0"
I20260619 13:08:00.661140 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:00.681522 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:00.685693 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:00.685913 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:00.704380 9177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34673
I20260619 13:08:00.704413 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34673 every 8 connection(s)
I20260619 13:08:00.706019 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:08:00.707360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9177
I20260619 13:08:00.708364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33485
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34673
--webserver_port=36169
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:08:00.716195 9244 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:08:00.719542 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Bootstrap starting.
I20260619 13:08:00.738456 9244 log.cc:824] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:00.747076 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:08:00.747642 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Bootstrap complete.
I20260619 13:08:00.756744 9244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:08:00.757573 9244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd4beb82db046a28b0403c14417a151, State: Initialized, Role: FOLLOWER
I20260619 13:08:00.758366 9244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:08:00.758571 9244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:08:00.758716 9244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:08:00.758903 9244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:08:00.762869 9244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:08:00.763384 9244 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd4beb82db046a28b0403c14417a151; no voters:
I20260619 13:08:00.764021 9244 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:08:00.764186 9248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:08:00.766059 9248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd4beb82db046a28b0403c14417a151, State: Running, Role: LEADER
I20260619 13:08:00.766572 9248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260619 13:08:00.767344 9244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:08:00.769312 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd4beb82db046a28b0403c14417a151. Latest consensus state: current_term: 2 leader_uuid: "4cd4beb82db046a28b0403c14417a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260619 13:08:00.769286 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4cd4beb82db046a28b0403c14417a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd4beb82db046a28b0403c14417a151" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260619 13:08:00.769747 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:00.769744 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:00.783257 9254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:08:00.788615 9254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:08:00.789985 9254 catalog_manager.cc:1259] Loaded cluster ID: 53546d7a2aad4ef2a1bf2fc6223fb69a
I20260619 13:08:00.790081 9254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:08:00.794106 9254 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:08:00.795521 9254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4cd4beb82db046a28b0403c14417a151: Loaded TSK: 0
I20260619 13:08:00.796972 9254 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:08:01.060846 9246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:01.061167 9246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:01.061347 9246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:01.070518 9246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:01.070684 9246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:01.070919 9246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:08:01.082001 9246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36169
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:01.083469 9246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:01.084884 9246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:01.100684 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:01.100921 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:01.100679 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:01.101589 9246 server_base.cc:1034] running on GCE node
I20260619 13:08:01.102267 9246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:01.103319 9246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:01.104542 9246 hybrid_clock.cc:648] HybridClock initialized: now 1781874481104479 us; error 55 us; skew 500 ppm
I20260619 13:08:01.104872 9246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:01.108410 9246 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:08:01.110294 9246 webserver.cc:460] Webserver started at http://127.0.97.1:36169/ using document root <none> and password file <none>
I20260619 13:08:01.111060 9246 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:01.111240 9246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:01.117394 9246 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:08:01.120551 9280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.122249 9246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:08:01.122466 9246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "265dd66b54d748c98e1f95c88fc8c830"
format_stamp: "Formatted at 2026-06-19 13:07:58 on dist-test-slave-q11v"
server_key: "8710748635d368732ec2238a8d697ac4"
server_key_iv: "ea75b5246de639f53722dc6b23ca697c"
server_key_version: "encryptionkey@0"
I20260619 13:08:01.123509 9246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:01.136476 9246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:01.140888 9246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:01.141173 9246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:01.142381 9246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:01.143754 9246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:01.143883 9246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.144016 9246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:01.144115 9246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.179942 9246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33485
I20260619 13:08:01.179999 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33485 every 8 connection(s)
I20260619 13:08:01.181546 9246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:08:01.183208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9246
I20260619 13:08:01.184029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42463
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34673
--webserver_port=32815
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:08:01.201771 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.184888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57393 (local address 127.0.97.62:34673)
0619 13:08:01.186120 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.186133 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.186775 (+ 642us) server_negotiation.cc:407] Connection header received
0619 13:08:01.188949 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.188963 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.189306 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.189718 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.191245 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.192491 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.194127 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.194805 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.195010 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.200766 (+ 5756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57393: BlockingRecv error: recv got EOF from 127.0.97.1:57393 (error 108)
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":817,"threads_started":1}
I20260619 13:08:01.201838 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.185128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:57393)
0619 13:08:01.186585 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.186622 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.188349 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.189873 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.189883 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.190499 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.191042 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.191059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.192658 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.192668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.193971 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.193984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.195040 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.195047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.195330 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.196053 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.196075 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.200430 (+ 4355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1036,"threads_started":1}
W20260619 13:08:01.202513 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34673 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34673: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:01.210840 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.203166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:53169)
0619 13:08:01.203318 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.203338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.203466 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.203844 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.203850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.204190 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.204339 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.204347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.205356 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.205362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.206338 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.206348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.207016 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.207021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.207199 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.207783 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.207807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.210511 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:08:01.210912 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.203386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53169 (local address 127.0.97.62:34673)
0619 13:08:01.203581 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.203587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.203610 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:08:01.203653 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.203657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.203709 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.203813 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.204477 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.205211 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.206474 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.206873 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.207004 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.210701 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53169: BlockingRecv error: recv got EOF from 127.0.97.1:53169 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:01.219656 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.211556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:48141)
0619 13:08:01.211713 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.211738 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.211822 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.212168 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.212171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.212433 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.212573 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.212580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.214084 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.214088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.215038 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.215049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.215967 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.215971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.216090 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.216722 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.216739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.219439 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:08:01.219957 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.211613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48141 (local address 127.0.97.62:34673)
0619 13:08:01.211770 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.211776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.211796 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:08:01.211931 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.211937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.212023 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.212130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.212698 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.213922 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.215177 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.215830 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.215954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.219694 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48141: BlockingRecv error: recv got EOF from 127.0.97.1:48141 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 13:08:01.220031 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:08:01.534010 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:01.534308 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:01.534478 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:01.542053 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:01.542171 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:01.542354 9397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:08:01.552335 9397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32815
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:01.553869 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:01.555233 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:01.568605 9402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:01.568779 9403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:01.569118 9405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:01.569628 9397 server_base.cc:1034] running on GCE node
I20260619 13:08:01.570238 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:01.571128 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:01.572397 9397 hybrid_clock.cc:648] HybridClock initialized: now 1781874481572334 us; error 52 us; skew 500 ppm
I20260619 13:08:01.572718 9397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:01.575879 9397 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:08:01.577368 9397 webserver.cc:460] Webserver started at http://127.0.97.2:32815/ using document root <none> and password file <none>
I20260619 13:08:01.578197 9397 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:01.578359 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:01.584410 9397 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 13:08:01.587324 9412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.588769 9397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:08:01.588955 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "838b8bb5032b4192a2120b49648765d4"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "401161c89978d39c975637ae4e6b2830"
server_key_iv: "7a99cc54ccb4ebb7811a867b95436e7e"
server_key_version: "encryptionkey@0"
I20260619 13:08:01.589991 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:01.605774 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:01.609766 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:01.610049 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:01.611057 9397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:01.612382 9397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:01.612514 9397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.612649 9397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:01.612743 9397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:01.646041 9397 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42463
I20260619 13:08:01.646096 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42463 every 8 connection(s)
I20260619 13:08:01.647504 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:08:01.649567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9397
I20260619 13:08:01.650341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45147
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34673
--webserver_port=37135
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:08:01.665037 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.650360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58359 (local address 127.0.97.62:34673)
0619 13:08:01.650610 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.650618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.651904 (+ 1286us) server_negotiation.cc:407] Connection header received
0619 13:08:01.653819 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.653828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.653932 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.654062 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.655543 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.656320 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.658704 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.659114 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.659209 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.664777 (+ 5568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58359: BlockingRecv error: recv got EOF from 127.0.97.2:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:08:01.665921 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.650866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:58359)
0619 13:08:01.651718 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.651768 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.653474 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.654260 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.654268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.654798 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.655348 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.655363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.656526 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.656534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.658539 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.658557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.659353 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.659362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.659690 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.660431 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.660464 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.664611 (+ 4147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
W20260619 13:08:01.666616 9525 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34673 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34673: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:01.674978 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.667284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:59817)
0619 13:08:01.667525 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.667545 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.667662 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.668011 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.668015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.668392 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.668525 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.668533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.669504 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.669509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.670459 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.670469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.671159 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.671164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.671338 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.671950 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.671966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.674702 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 13:08:01.675135 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.667348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59817 (local address 127.0.97.62:34673)
0619 13:08:01.667537 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.667544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.667596 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:08:01.667782 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.667787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.667859 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.667989 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.668659 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.669363 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.670596 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.671005 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.671150 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.674905 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59817: BlockingRecv error: recv got EOF from 127.0.97.2:59817 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:01 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:01.684463 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:01.675772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:33771)
0619 13:08:01.676007 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:01.676025 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:01.676124 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:01.676406 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:01.676411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:01.676771 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:01.676952 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.676962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.678060 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.678065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.679594 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:01.679612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.680303 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.680310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:01.680491 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.681083 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:01.681105 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:01.684157 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 13:08:01.684563 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:01.675867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33771 (local address 127.0.97.62:34673)
0619 13:08:01.676026 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:01.676030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:01.676093 (+ 63us) server_negotiation.cc:407] Connection header received
0619 13:08:01.676219 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:01.676222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:01.676273 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:01.676350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:01.677089 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.677931 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.679734 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:01.680157 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:01.680248 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:01.684349 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33771: BlockingRecv error: recv got EOF from 127.0.97.2:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 13:08:01.684839 9525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:08:01.991267 9528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:01.991632 9528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:01.991827 9528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:01.999341 9528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:01.999460 9528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:01.999573 9528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:08:02.009402 9528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37135
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:02.010730 9528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:02.011961 9528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:02.025051 9533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:02.025010 9536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:02.025074 9534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:02.025655 9528 server_base.cc:1034] running on GCE node
I20260619 13:08:02.026252 9528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:02.027602 9528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:02.028832 9528 hybrid_clock.cc:648] HybridClock initialized: now 1781874482028805 us; error 62 us; skew 500 ppm
I20260619 13:08:02.029178 9528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:02.032949 9528 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:08:02.034564 9528 webserver.cc:460] Webserver started at http://127.0.97.3:37135/ using document root <none> and password file <none>
I20260619 13:08:02.035277 9528 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:02.035444 9528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:02.040989 9528 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 13:08:02.043998 9543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:02.045470 9528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 13:08:02.045718 9528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "4568e987c34c4fb3908c6723decac591"
format_stamp: "Formatted at 2026-06-19 13:07:59 on dist-test-slave-q11v"
server_key: "b7216a844e77482245a9ef14ed309290"
server_key_iv: "9cdd1fc7af3272dbd4846ffbef6aba06"
server_key_version: "encryptionkey@0"
I20260619 13:08:02.046730 9528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:02.057534 9528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:02.061836 9528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:02.062112 9528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:02.063082 9528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:02.064512 9528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:02.064638 9528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:02.064760 9528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:02.064857 9528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:02.100135 9528 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45147
I20260619 13:08:02.100226 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45147 every 8 connection(s)
I20260619 13:08:02.101580 9528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:08:02.104728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9528
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:08:02.120680 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:02.104228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50973 (local address 127.0.97.62:34673)
0619 13:08:02.104410 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.104415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.106015 (+ 1600us) server_negotiation.cc:407] Connection header received
0619 13:08:02.109642 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.109652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.109822 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.109945 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.111531 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.112230 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.113828 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.114320 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.114474 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.120429 (+ 5955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50973: BlockingRecv error: recv got EOF from 127.0.97.3:50973 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:08:02.121510 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.104882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:50973)
0619 13:08:02.105807 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:02.105888 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:02.109187 (+ 3299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:02.110280 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:02.110290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:02.110798 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:02.111360 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.111377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.112393 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.112398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.113691 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.113702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.114580 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.114587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.114953 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.115703 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:02.115736 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:02.120271 (+ 4535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":354,"threads_started":1}
W20260619 13:08:02.122299 9656 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34673 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34673: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:02.130612 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.105853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.97.62:34673)
0619 13:08:02.110112 (+ 4259us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.110121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.114195 (+ 4074us) server_negotiation.cc:407] Connection header received
0619 13:08:02.114320 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.114326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.114406 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.114517 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.116550 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.117563 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.119124 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.119810 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.119945 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.123068 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:02.123082 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:02.123104 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:02.123153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:02.126892 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:02.127830 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:02.127841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:02.127857 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:02.128092 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:02.128528 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:02.128532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:02.128534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:02.129105 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:02.129327 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:02.129960 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 13:08:02.130316 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4104,"thread_start_us":3946,"threads_started":1}
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:02.132066 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.122947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:35685)
0619 13:08:02.123128 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:02.123148 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:02.123260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:02.123832 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:02.123836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:02.124286 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:02.124440 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.124448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.126054 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.126059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.127078 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.127095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.127875 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.127880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.128003 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.128626 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:02.128644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:02.131787 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 13:08:02.132206 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:02.123045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35685 (local address 127.0.97.62:34673)
0619 13:08:02.123529 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.123535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.123556 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:08:02.123606 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.123611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.123692 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.123799 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.124584 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.125771 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.127223 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.127707 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.127863 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.131975 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35685: BlockingRecv error: recv got EOF from 127.0.97.3:35685 (error 108)
Metrics: {"server-negotiator.queue_time_us":359}
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:02.140710 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.133080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:47741)
0619 13:08:02.133296 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:02.133315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:02.133411 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:02.133810 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:02.133815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:02.134165 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:02.134309 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.134317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.135472 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.135478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.136365 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.136376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.137056 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.137061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.137242 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.137848 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:02.137864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:02.140405 (+ 2541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 13:08:02.140846 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:02.133217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47741 (local address 127.0.97.62:34673)
0619 13:08:02.133494 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.133500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.133516 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:08:02.133562 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.133566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.133657 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.133761 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.134466 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.135334 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.136511 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.136912 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.137047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.140585 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47741: BlockingRecv error: recv got EOF from 127.0.97.3:47741 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
W20260619 13:08:02.141127 9656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:02.229408 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.220823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:43685)
0619 13:08:02.221353 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:02.221370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:02.221526 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:02.221912 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:02.221916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:02.222401 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:02.222587 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.222595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.223611 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.223617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.224586 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.224597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.225374 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.225381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.225559 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.226230 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:02.226249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:02.229158 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
W20260619 13:08:02.229848 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:02.220891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43685 (local address 127.0.97.62:34673)
0619 13:08:02.221095 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.221103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.221462 (+ 359us) server_negotiation.cc:407] Connection header received
0619 13:08:02.221696 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.221701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.221771 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.221882 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.222739 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.223459 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.224732 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.225148 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.225309 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.229598 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43685: BlockingRecv error: recv got EOF from 127.0.97.1:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:02 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:02.700964 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:02.685448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:55477)
0619 13:08:02.685967 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:02.685988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:02.686143 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:02.686485 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:02.686489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:02.686999 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:02.687200 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.687210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.688217 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.688223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.689226 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:02.689237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.690035 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.690042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:02.690215 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.697518 (+ 7303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:02.697538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:02.700733 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":292,"threads_started":1}
W20260619 13:08:02.701200 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:02.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55477 (local address 127.0.97.62:34673)
0619 13:08:02.685783 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:02.685790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:02.686077 (+ 287us) server_negotiation.cc:407] Connection header received
0619 13:08:02.686273 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:02.686280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:02.686345 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:02.686450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:02.687353 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.688063 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.689382 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:02.689857 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:02.690000 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:02.700939 (+ 10939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55477: BlockingRecv error: recv got EOF from 127.0.97.2:55477 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:03.150884 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:03.141908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:40039)
0619 13:08:03.142497 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:03.142521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:03.142689 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:03.143091 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:03.143095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:03.143637 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:03.143850 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.143884 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.144961 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.144965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.145920 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.145932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.146665 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.146669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.146808 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.147363 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:03.147379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:03.150589 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":328,"threads_started":1}
W20260619 13:08:03.151114 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:03.141985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40039 (local address 127.0.97.62:34673)
0619 13:08:03.142251 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:03.142260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:03.142620 (+ 360us) server_negotiation.cc:407] Connection header received
0619 13:08:03.142828 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:03.142834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:03.142942 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:03.143053 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:03.144042 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.144824 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.146058 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.146509 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.146612 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.150852 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40039: BlockingRecv error: recv got EOF from 127.0.97.3:40039 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:03.238839 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:03.230316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:60117)
0619 13:08:03.230847 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:03.230862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:03.231014 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:03.231367 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:03.231372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:03.231808 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:03.231947 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.231955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.232969 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.232975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.233940 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.233951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.234680 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.234686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.234870 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.235516 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:03.235534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:03.238423 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
W20260619 13:08:03.238881 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:03.230482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60117 (local address 127.0.97.62:34673)
0619 13:08:03.230710 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:03.230716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:03.230946 (+ 230us) server_negotiation.cc:407] Connection header received
0619 13:08:03.231148 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:03.231154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:03.231231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:03.231336 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:03.232094 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.232829 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.234092 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.234515 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.234667 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.238626 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60117: BlockingRecv error: recv got EOF from 127.0.97.1:60117 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:03 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:03.711403 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:03.701920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:53527)
0619 13:08:03.702458 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:03.702479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:03.702624 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:03.703071 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:03.703076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:03.703509 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:03.703722 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.703733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.704913 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.704919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.706418 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:03.706433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.707143 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.707149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:03.707320 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.707972 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:03.707995 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:03.711177 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":315,"threads_started":1}
W20260619 13:08:03.711683 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:03.701972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53527 (local address 127.0.97.62:34673)
0619 13:08:03.702188 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:03.702195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:03.702569 (+ 374us) server_negotiation.cc:407] Connection header received
0619 13:08:03.702788 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:03.702792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:03.702891 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:03.703003 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:03.703858 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.704769 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.706559 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:03.706993 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:03.707095 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:03.711405 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53527: BlockingRecv error: recv got EOF from 127.0.97.2:53527 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:04.165222 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:04.151797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:42821)
0619 13:08:04.158009 (+ 6212us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:04.158028 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:04.158154 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:04.158444 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:04.158447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:04.158839 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:04.158978 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.158986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.159992 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.159997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.160893 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.160905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.161531 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.161535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.161728 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.162275 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:04.162292 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:04.164899 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6105,"thread_start_us":5983,"threads_started":1}
W20260619 13:08:04.165364 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:04.151947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42821 (local address 127.0.97.62:34673)
0619 13:08:04.152221 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:04.152227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:04.158095 (+ 5868us) server_negotiation.cc:407] Connection header received
0619 13:08:04.158249 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:04.158253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:04.158318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:04.158403 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:04.159113 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.159856 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.161010 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.161407 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.161496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.165086 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42821: BlockingRecv error: recv got EOF from 127.0.97.3:42821 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:04.248744 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:04.239980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:48791)
0619 13:08:04.240481 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:04.240498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:04.240673 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:04.241100 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:04.241106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:04.241576 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:04.241836 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.241849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.243075 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.243081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.243976 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.243987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.244687 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.244693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.244890 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.245503 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:04.245523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:04.248387 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
W20260619 13:08:04.248852 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:04.240053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48791 (local address 127.0.97.62:34673)
0619 13:08:04.240278 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:04.240286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:04.240594 (+ 308us) server_negotiation.cc:407] Connection header received
0619 13:08:04.240855 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:04.240859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:04.240929 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:04.241055 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:04.241968 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.242926 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.244127 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.244544 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.244676 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.248601 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48791: BlockingRecv error: recv got EOF from 127.0.97.1:48791 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:04 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:04.720973 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:04.712452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:56817)
0619 13:08:04.712938 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:04.712958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:04.713109 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:04.713438 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:04.713442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:04.713931 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:04.714161 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.714174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.715571 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.715577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.716475 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:04.716487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.717132 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.717136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:04.717268 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.717897 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:04.717914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:04.720712 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":252,"threads_started":1}
W20260619 13:08:04.721108 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:04.712519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56817 (local address 127.0.97.62:34673)
0619 13:08:04.712746 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:04.712755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:04.713053 (+ 298us) server_negotiation.cc:407] Connection header received
0619 13:08:04.713211 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:04.713215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:04.713277 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:04.713360 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:04.714330 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.715381 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.716609 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:04.716999 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:04.717099 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:04.720893 (+ 3794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56817: BlockingRecv error: recv got EOF from 127.0.97.2:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:05.174525 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:05.166057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:35879)
0619 13:08:05.166497 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:05.166514 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:05.166647 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:05.166930 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:05.166933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:05.167263 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:05.167427 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.167435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.168485 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.168489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.169476 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.169488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.170228 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.170234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.170403 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.171024 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:05.171041 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:05.174205 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
W20260619 13:08:05.174762 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:05.166170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35879 (local address 127.0.97.62:34673)
0619 13:08:05.166441 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:05.166448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:05.166629 (+ 181us) server_negotiation.cc:407] Connection header received
0619 13:08:05.166745 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:05.166749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:05.166810 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:05.166892 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:05.167615 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.168353 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.169626 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.170033 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.170183 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.174539 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35879: BlockingRecv error: recv got EOF from 127.0.97.3:35879 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:05.257951 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:05.249703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:39249)
0619 13:08:05.250201 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:05.250218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:05.250380 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:05.250750 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:05.250755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:05.251160 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:05.251313 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.251322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.252314 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.252321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.253275 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.253286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.254051 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.254057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.254258 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.254896 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:05.254913 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:05.257567 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
W20260619 13:08:05.258080 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:05.249747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39249 (local address 127.0.97.62:34673)
0619 13:08:05.250076 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:05.250082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:05.250311 (+ 229us) server_negotiation.cc:407] Connection header received
0619 13:08:05.250504 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:05.250510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:05.250585 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:05.250722 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:05.251461 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.252184 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.253418 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.253902 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.254047 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.257799 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39249: BlockingRecv error: recv got EOF from 127.0.97.1:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:05 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:05.730407 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:05.721896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:52371)
0619 13:08:05.722449 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:05.722492 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:05.722614 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:05.722894 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:05.722897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:05.723237 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:05.723378 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.723385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.724284 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.724287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.725826 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:05.725841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.726525 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.726532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:05.726676 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.727224 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:05.727240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:05.730074 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":361,"threads_started":1}
W20260619 13:08:05.730515 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:05.721971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52371 (local address 127.0.97.62:34673)
0619 13:08:05.722154 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:05.722158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:05.722571 (+ 413us) server_negotiation.cc:407] Connection header received
0619 13:08:05.722713 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:05.722716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:05.722775 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:05.722853 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:05.723489 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.724173 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.725947 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:05.726375 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:05.726484 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:05.730276 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52371: BlockingRecv error: recv got EOF from 127.0.97.2:52371 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:06.184062 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:06.175688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:57951)
0619 13:08:06.176263 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:06.176279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:06.176406 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:06.176770 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:06.176775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:06.177167 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:06.177324 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.177333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.178441 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.178449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.179396 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.179407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.180134 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.180139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.180328 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.181012 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:06.181030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:06.183815 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":349,"threads_started":1}
W20260619 13:08:06.184280 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:06.175752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57951 (local address 127.0.97.62:34673)
0619 13:08:06.175926 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:06.175934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:06.176340 (+ 406us) server_negotiation.cc:407] Connection header received
0619 13:08:06.176546 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:06.176553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:06.176631 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:06.176738 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:06.177468 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.178302 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.179533 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.179980 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.180124 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.184025 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57951: BlockingRecv error: recv got EOF from 127.0.97.3:57951 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:06.267434 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:06.259031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:52493)
0619 13:08:06.259632 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:06.259649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:06.259796 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:06.260167 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:06.260172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:06.260587 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:06.260746 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.260755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.261805 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.261812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.262744 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.262755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.263467 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.263473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.263647 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.264210 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:06.264227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:06.267104 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":309,"threads_started":1}
W20260619 13:08:06.267547 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:06.259112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52493 (local address 127.0.97.62:34673)
0619 13:08:06.259321 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:06.259329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:06.259738 (+ 409us) server_negotiation.cc:407] Connection header received
0619 13:08:06.259953 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:06.259958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:06.260023 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:06.260130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:06.260899 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.261677 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.262894 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.263316 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.263448 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.267337 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52493: BlockingRecv error: recv got EOF from 127.0.97.1:52493 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:06 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:06.740113 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:06.731459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:40723)
0619 13:08:06.731930 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:06.731951 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:06.732066 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:06.732354 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:06.732357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:06.732751 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:06.732915 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.732923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.734491 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.734497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.735630 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:06.735645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.736355 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.736359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:06.736510 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.737159 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:06.737177 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:06.739869 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
W20260619 13:08:06.740427 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:06.731665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40723 (local address 127.0.97.62:34673)
0619 13:08:06.731909 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:06.731918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:06.732026 (+ 108us) server_negotiation.cc:407] Connection header received
0619 13:08:06.732163 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:06.732167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:06.732227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:06.732319 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:06.733076 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.734287 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.735785 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:06.736209 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:06.736315 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:06.740113 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40723: BlockingRecv error: recv got EOF from 127.0.97.2:40723 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:07.193835 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:07.185053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:37405)
0619 13:08:07.185669 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:07.185690 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:07.185833 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:07.186228 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:07.186234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:07.186754 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:07.186935 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.186944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.188094 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.188098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.189016 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.189027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.189715 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.189719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.189838 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.190458 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:07.190478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:07.193479 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":305,"threads_started":1}
W20260619 13:08:07.193892 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:07.185162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37405 (local address 127.0.97.62:34673)
0619 13:08:07.185347 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:07.185356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:07.185799 (+ 443us) server_negotiation.cc:407] Connection header received
0619 13:08:07.185961 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:07.185965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:07.186068 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:07.186196 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:07.187117 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.187968 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.189132 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.189530 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.189694 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.193692 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37405: BlockingRecv error: recv got EOF from 127.0.97.3:37405 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:07.277211 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:07.268392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:52525)
0619 13:08:07.268953 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:07.268979 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:07.269095 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:07.269509 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:07.269513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:07.270032 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:07.270228 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.270238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.271617 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.271621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.272583 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.272595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.273363 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.273367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.273495 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.274110 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:07.274127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:07.276989 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
W20260619 13:08:07.277499 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:07.268442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52525 (local address 127.0.97.62:34673)
0619 13:08:07.268607 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:07.268611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:07.269058 (+ 447us) server_negotiation.cc:407] Connection header received
0619 13:08:07.269232 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:07.269236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:07.269312 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:07.269405 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:07.270387 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.271463 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.272761 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.273206 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.273314 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.277200 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52525: BlockingRecv error: recv got EOF from 127.0.97.1:52525 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:07 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:07.750038 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:07.741191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:37103)
0619 13:08:07.741713 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:07.741735 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:07.741886 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:07.742236 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:07.742241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:07.742662 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:07.742814 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.742823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.743962 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.743966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.745209 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:07.745220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.746161 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.746167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:07.746342 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.746995 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:07.747011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:07.749784 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
W20260619 13:08:07.750407 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:07.741274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37103 (local address 127.0.97.62:34673)
0619 13:08:07.741505 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:07.741520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:07.741807 (+ 287us) server_negotiation.cc:407] Connection header received
0619 13:08:07.742025 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:07.742029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:07.742090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:07.742209 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:07.742993 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.743786 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.745357 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:07.746003 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:07.746127 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:07.750135 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37103: BlockingRecv error: recv got EOF from 127.0.97.2:37103 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:08.204342 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:08.194945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:60923)
0619 13:08:08.195440 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:08.195457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:08.195576 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:08.195902 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:08.195905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:08.196270 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:08.196435 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.196443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.197685 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.197690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.199093 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.199110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.200072 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.200077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.200231 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.200898 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:08.200916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:08.204111 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
W20260619 13:08:08.204710 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:08.195056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60923 (local address 127.0.97.62:34673)
0619 13:08:08.195299 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:08.195305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:08.195567 (+ 262us) server_negotiation.cc:407] Connection header received
0619 13:08:08.195697 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:08.195710 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:08.195772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:08.195862 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:08.196596 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.197488 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.199283 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.199868 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.199978 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.204439 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60923: BlockingRecv error: recv got EOF from 127.0.97.3:60923 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:08.287218 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:08.278147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:46737)
0619 13:08:08.278801 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:08.278819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:08.278939 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:08.279227 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:08.279230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:08.279642 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:08.279831 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.279841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.280896 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.280900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.281849 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.281860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.282639 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.282662 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.282843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.283563 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:08.283582 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:08.286982 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":311,"threads_started":1}
W20260619 13:08:08.287487 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:08.278273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46737 (local address 127.0.97.62:34673)
0619 13:08:08.278456 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:08.278460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:08.278893 (+ 433us) server_negotiation.cc:407] Connection header received
0619 13:08:08.279030 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:08.279034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:08.279095 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:08.279190 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:08.279985 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.280760 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.281996 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.282415 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.282547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.287183 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46737: BlockingRecv error: recv got EOF from 127.0.97.1:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:08 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:08.760159 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:08.751017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:56869)
0619 13:08:08.751685 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:08.751718 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:08.751852 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:08.752174 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:08.752177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:08.752561 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:08.752815 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.752826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.754048 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.754052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.754972 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:08.754984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.755689 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.755694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:08.755816 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.756414 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:08.756430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:08.759815 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":305,"threads_started":1}
W20260619 13:08:08.760257 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:08.751037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56869 (local address 127.0.97.62:34673)
0619 13:08:08.751231 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:08.751235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:08.751806 (+ 571us) server_negotiation.cc:407] Connection header received
0619 13:08:08.751959 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:08.751963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:08.752043 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:08.752128 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:08.752983 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.753888 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.755112 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:08.755549 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:08.755662 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:08.760006 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56869: BlockingRecv error: recv got EOF from 127.0.97.2:56869 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:09.214989 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:09.205434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:40683)
0619 13:08:09.206095 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:09.206113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:09.206252 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:09.206612 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:09.206616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:09.207038 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:09.207212 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.207222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.208533 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.208537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.210065 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.210081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.210855 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.210860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.210988 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.211626 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:09.211649 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:09.214743 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":330,"threads_started":1}
W20260619 13:08:09.215154 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:09.205642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40683 (local address 127.0.97.62:34673)
0619 13:08:09.205882 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:09.205889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:09.206237 (+ 348us) server_negotiation.cc:407] Connection header received
0619 13:08:09.206352 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:09.206356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:09.206423 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:09.206518 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:09.207379 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.208320 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.210208 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.210652 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.210759 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.214959 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40683: BlockingRecv error: recv got EOF from 127.0.97.3:40683 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:09.297713 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:09.288325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:58947)
0619 13:08:09.288837 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:09.288854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:09.288977 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:09.289389 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:09.289392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:09.289915 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:09.290090 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.290100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.291161 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.291166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.292134 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.292147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.292967 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.292973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.293180 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.294031 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:09.294056 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:09.297416 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
W20260619 13:08:09.297899 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:09.288580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58947 (local address 127.0.97.62:34673)
0619 13:08:09.288806 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:09.288815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:09.288933 (+ 118us) server_negotiation.cc:407] Connection header received
0619 13:08:09.289114 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:09.289119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:09.289208 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:09.289301 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:09.290249 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.290997 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.292289 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.292750 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.292851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.297661 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58947: BlockingRecv error: recv got EOF from 127.0.97.1:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:09 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:09.770282 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:09.761290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:60837)
0619 13:08:09.761805 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:09.761824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:09.761982 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:09.762305 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:09.762309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:09.762735 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:09.762878 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.762886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.764042 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.764048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.764999 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:09.765009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.766100 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.766129 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:09.766335 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.767083 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:09.767129 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:09.770050 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
W20260619 13:08:09.770419 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:09.761365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60837 (local address 127.0.97.62:34673)
0619 13:08:09.761599 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:09.761607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:09.761912 (+ 305us) server_negotiation.cc:407] Connection header received
0619 13:08:09.762104 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:09.762115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:09.762175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:09.762279 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:09.763019 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.763892 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.765134 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:09.765764 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:09.765899 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:09.770227 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60837: BlockingRecv error: recv got EOF from 127.0.97.2:60837 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:10.224540 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:10.216081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:35509)
0619 13:08:10.216652 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:10.216679 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:10.216797 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:10.217206 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:10.217210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:10.217840 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:10.218054 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.218069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.219223 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.219227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.220101 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.220120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.220778 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.220782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.220894 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.221489 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:10.221505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:10.224323 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
W20260619 13:08:10.224793 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:10.216161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35509 (local address 127.0.97.62:34673)
0619 13:08:10.216421 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:10.216430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:10.216776 (+ 346us) server_negotiation.cc:407] Connection header received
0619 13:08:10.216934 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:10.216940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:10.217033 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:10.217144 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:10.218242 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.219084 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.220230 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.220633 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.220724 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.224522 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35509: BlockingRecv error: recv got EOF from 127.0.97.3:35509 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:10.307134 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:10.298781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:46047)
0619 13:08:10.299264 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:10.299280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:10.299452 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:10.299870 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:10.299875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:10.300311 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:10.300474 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.300483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.301855 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.301859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.302881 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.302894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.303596 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.303600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.303738 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.304265 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:10.304282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:10.306897 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":231,"threads_started":1}
W20260619 13:08:10.307312 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:10.298793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46047 (local address 127.0.97.62:34673)
0619 13:08:10.299007 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:10.299015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:10.299371 (+ 356us) server_negotiation.cc:407] Connection header received
0619 13:08:10.299605 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:10.299609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:10.299726 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:10.299838 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:10.300643 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.301699 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.303005 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.303469 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.303573 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.307084 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46047: BlockingRecv error: recv got EOF from 127.0.97.1:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:10 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:10.780284 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:10.771312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:60559)
0619 13:08:10.772281 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:10.772308 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:10.772460 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:10.772876 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:10.772884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:10.773438 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:10.773726 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.773737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.774765 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.774770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.775686 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:10.775697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.776347 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.776353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:10.776483 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.777121 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:10.777138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:10.780030 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":538,"threads_started":1}
W20260619 13:08:10.780452 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:10.771683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60559 (local address 127.0.97.62:34673)
0619 13:08:10.771908 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:10.771915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:10.772400 (+ 485us) server_negotiation.cc:407] Connection header received
0619 13:08:10.772604 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:10.772609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:10.772699 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:10.772792 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:10.773867 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.774623 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.775807 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:10.776206 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:10.776300 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:10.780224 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60559: BlockingRecv error: recv got EOF from 127.0.97.2:60559 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:11.234412 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:11.225570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:60747)
0619 13:08:11.226133 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:11.226155 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:11.226316 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:11.226748 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:11.226753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:11.227333 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:11.227594 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.227607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.228811 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.228816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.229792 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.229805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.230521 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.230526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.230661 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.231330 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:11.231348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:11.234177 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":262,"threads_started":1}
W20260619 13:08:11.234613 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:11.225701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60747 (local address 127.0.97.62:34673)
0619 13:08:11.225937 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:11.225943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:11.226265 (+ 322us) server_negotiation.cc:407] Connection header received
0619 13:08:11.226455 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:11.226460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:11.226581 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:11.226667 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:11.227754 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.228668 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.229932 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.230354 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.230454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.234390 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60747: BlockingRecv error: recv got EOF from 127.0.97.3:60747 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:11.316381 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:11.308217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:32899)
0619 13:08:11.308759 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:11.308775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:11.308886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:11.309184 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:11.309187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:11.309545 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:11.309733 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.309740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.310768 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.310774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.311692 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.311703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.312475 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.312481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.312666 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.313284 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:11.313304 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:11.316153 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":321,"threads_started":1}
W20260619 13:08:11.316608 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:11.308337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32899 (local address 127.0.97.62:34673)
0619 13:08:11.308536 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:11.308540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:11.308851 (+ 311us) server_negotiation.cc:407] Connection header received
0619 13:08:11.308984 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:11.308988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:11.309050 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:11.309141 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:11.309868 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.310609 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.311843 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.312316 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.312453 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.316335 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:32899: BlockingRecv error: recv got EOF from 127.0.97.1:32899 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:11 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:11.790282 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:11.781371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:40863)
0619 13:08:11.781948 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:11.781973 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:11.782089 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:11.782391 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:11.782394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:11.782772 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:11.782941 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.782951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.784063 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.784067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.785053 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:11.785066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.785784 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.785789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:11.785920 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.786671 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:11.786699 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:11.789950 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":355,"threads_started":1}
W20260619 13:08:11.790390 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:11.781413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40863 (local address 127.0.97.62:34673)
0619 13:08:11.781662 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:11.781672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:11.782051 (+ 379us) server_negotiation.cc:407] Connection header received
0619 13:08:11.782184 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:11.782187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:11.782247 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:11.782329 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:11.783152 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.783893 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.785201 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:11.785640 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:11.785733 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:11.790157 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40863: BlockingRecv error: recv got EOF from 127.0.97.2:40863 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:12.243505 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:12.235363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:33949)
0619 13:08:12.235843 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:12.235860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:12.235985 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:12.236288 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:12.236298 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:12.236667 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:12.236809 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.236817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.237977 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.237981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.238934 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.238946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.239665 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.239670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.239795 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.240381 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:12.240398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:12.243198 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":239,"threads_started":1}
W20260619 13:08:12.243623 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:12.235565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33949 (local address 127.0.97.62:34673)
0619 13:08:12.235865 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:12.235869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:12.235957 (+ 88us) server_negotiation.cc:407] Connection header received
0619 13:08:12.236084 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:12.236087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:12.236163 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:12.236248 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:12.236971 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.237790 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.239082 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.239496 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.239598 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.243367 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33949: BlockingRecv error: recv got EOF from 127.0.97.3:33949 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:12.325836 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:12.317424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:36395)
0619 13:08:12.317985 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:12.318014 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:12.318136 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:12.318435 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:12.318438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:12.318849 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:12.319040 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.319050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.320017 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.320021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.320990 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.321002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.321860 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.321865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.321991 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.322593 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:12.322611 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:12.325431 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":311,"threads_started":1}
W20260619 13:08:12.325912 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:12.317505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36395 (local address 127.0.97.62:34673)
0619 13:08:12.317726 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:12.317730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:12.318094 (+ 364us) server_negotiation.cc:407] Connection header received
0619 13:08:12.318236 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:12.318239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:12.318308 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:12.318392 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:12.319163 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.319862 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.321147 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.321677 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.321819 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.325671 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36395: BlockingRecv error: recv got EOF from 127.0.97.1:36395 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:12 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:12.810144 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:12.791210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:37107)
0619 13:08:12.791726 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:12.791746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:12.791870 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:12.792159 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:12.792162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:12.792529 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:12.792703 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.792711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.793947 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.793952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.795008 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:12.795020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.795866 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.795872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:12.795990 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.806393 (+ 10403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:12.806415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:12.809899 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":244,"threads_started":1}
W20260619 13:08:12.810318 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:12.791394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37107 (local address 127.0.97.62:34673)
0619 13:08:12.791597 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:12.791603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:12.791839 (+ 236us) server_negotiation.cc:407] Connection header received
0619 13:08:12.791973 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:12.791976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:12.792037 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:12.792130 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:12.792914 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.793767 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.795141 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:12.795643 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:12.795740 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:12.810101 (+ 14361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37107: BlockingRecv error: recv got EOF from 127.0.97.2:37107 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:13.253485 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:13.244593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:52701)
0619 13:08:13.245124 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:13.245139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:13.245311 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:13.245693 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:13.245698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:13.246117 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:13.246298 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.246306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.247304 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.247312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.248374 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.248389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.249157 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.249163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.249342 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.250099 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:13.250116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:13.253187 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":300,"threads_started":1}
W20260619 13:08:13.253636 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:13.244694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52701 (local address 127.0.97.62:34673)
0619 13:08:13.244946 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:13.244953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:13.245238 (+ 285us) server_negotiation.cc:407] Connection header received
0619 13:08:13.245440 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:13.245445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:13.245516 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:13.245635 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:13.246432 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.247169 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.248551 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.248996 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.249130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.253402 (+ 4272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52701: BlockingRecv error: recv got EOF from 127.0.97.3:52701 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:13.335853 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:13.327008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:57721)
0619 13:08:13.327553 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:13.327576 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:13.327791 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:13.328270 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:13.328275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:13.328775 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:13.328943 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.328951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.330040 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.330048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.330958 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.330969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.331731 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.331737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.331923 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.332610 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:13.332628 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:13.335575 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
W20260619 13:08:13.336035 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:13.327176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57721 (local address 127.0.97.62:34673)
0619 13:08:13.327429 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:13.327458 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:13.327680 (+ 222us) server_negotiation.cc:407] Connection header received
0619 13:08:13.327963 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:13.327969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:13.328081 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:13.328195 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:13.329104 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.329893 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.331098 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.331586 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.331723 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.335793 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57721: BlockingRecv error: recv got EOF from 127.0.97.1:57721 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:13 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:13.819743 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:13.811130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:43685)
0619 13:08:13.811688 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:13.811713 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:13.811880 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:13.812238 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:13.812242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:13.812633 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:13.812780 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.812788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.813888 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.813894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.814812 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:13.814823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.815543 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.815549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:13.815729 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.816356 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:13.816375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:13.819497 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":309,"threads_started":1}
W20260619 13:08:13.820043 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:13.811192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43685 (local address 127.0.97.62:34673)
0619 13:08:13.811484 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:13.811492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:13.811809 (+ 317us) server_negotiation.cc:407] Connection header received
0619 13:08:13.812010 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:13.812017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:13.812081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:13.812188 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:13.812936 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.813745 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.814953 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:13.815390 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:13.815518 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:13.819757 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43685: BlockingRecv error: recv got EOF from 127.0.97.2:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:14.263053 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:14.254527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:46859)
0619 13:08:14.255043 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:14.255059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:14.255231 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:14.255564 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:14.255568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:14.255972 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:14.256135 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.256144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.257191 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.257196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.258154 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.258167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.258922 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.258928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.259115 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.259754 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:14.259771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:14.262718 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
W20260619 13:08:14.263182 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:14.254585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46859 (local address 127.0.97.62:34673)
0619 13:08:14.254800 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:14.254808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:14.255150 (+ 342us) server_negotiation.cc:407] Connection header received
0619 13:08:14.255362 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:14.255369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:14.255437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:14.255538 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:14.256288 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.257033 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.258315 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.258760 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.258907 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.262948 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46859: BlockingRecv error: recv got EOF from 127.0.97.3:46859 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:14.346400 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:14.337044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:35633)
0619 13:08:14.337744 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:14.337767 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:14.337940 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:14.338403 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:14.338408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:14.338962 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:14.339202 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.339213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.340552 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.340556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.341484 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.341495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.342309 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.342314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.342437 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.343028 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:14.343046 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:14.345962 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":310,"threads_started":1}
W20260619 13:08:14.346467 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:14.337222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35633 (local address 127.0.97.62:34673)
0619 13:08:14.337497 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:14.337504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:14.337868 (+ 364us) server_negotiation.cc:407] Connection header received
0619 13:08:14.338092 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:14.338097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:14.338246 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:14.338335 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:14.339375 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.340400 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.341643 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.342102 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.342219 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.346197 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35633: BlockingRecv error: recv got EOF from 127.0.97.1:35633 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:14 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:14.829071 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:14.820770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:36275)
0619 13:08:14.821313 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:14.821333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:14.821442 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:14.821778 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:14.821781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:14.822175 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:14.822331 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.822339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.823357 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.823360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.824370 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:14.824382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.825070 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.825075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:14.825206 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.825804 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:14.825822 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:14.828685 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":309,"threads_started":1}
W20260619 13:08:14.829195 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:14.820895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36275 (local address 127.0.97.62:34673)
0619 13:08:14.821066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:14.821070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:14.821412 (+ 342us) server_negotiation.cc:407] Connection header received
0619 13:08:14.821538 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:14.821541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:14.821652 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:14.821742 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:14.822478 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.823224 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.824499 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:14.824941 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:14.825035 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:14.828930 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36275: BlockingRecv error: recv got EOF from 127.0.97.2:36275 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:15.272634 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:15.264213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:53113)
0619 13:08:15.264796 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:15.264813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:15.264927 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:15.265259 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:15.265262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:15.265708 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:15.265890 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.265899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.266874 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.266877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.267854 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.267866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.268628 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.268632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.268746 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.269331 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:15.269347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:15.272375 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":311,"threads_started":1}
W20260619 13:08:15.272792 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:15.264316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53113 (local address 127.0.97.62:34673)
0619 13:08:15.264475 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:15.264479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:15.264900 (+ 421us) server_negotiation.cc:407] Connection header received
0619 13:08:15.265029 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:15.265033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:15.265094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:15.265195 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:15.266016 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.266761 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.268021 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.268460 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.268566 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.272574 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53113: BlockingRecv error: recv got EOF from 127.0.97.3:53113 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:15.355743 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:15.347457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:56361)
0619 13:08:15.347959 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:15.347976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:15.348090 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:15.348403 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:15.348407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:15.348780 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:15.348952 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.348960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.350064 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.350069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.351151 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.351163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.351894 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.351898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.352041 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.352624 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:15.352642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:15.355513 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":281,"threads_started":1}
W20260619 13:08:15.355945 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:15.347523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56361 (local address 127.0.97.62:34673)
0619 13:08:15.347777 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:15.347781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:15.348063 (+ 282us) server_negotiation.cc:407] Connection header received
0619 13:08:15.348183 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:15.348187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:15.348262 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:15.348349 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:15.349091 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.349892 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.351278 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.351754 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.351849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.355688 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56361: BlockingRecv error: recv got EOF from 127.0.97.1:56361 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:15 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:15.838613 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:15.830140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:47239)
0619 13:08:15.830677 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:15.830704 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:15.830879 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:15.831224 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:15.831229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:15.831688 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:15.831868 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.831877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.832978 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.832984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.833955 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:15.833966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.834705 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.834710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:15.834902 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.835573 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:15.835590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:15.838286 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":254,"threads_started":1}
W20260619 13:08:15.838686 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:15.830251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47239 (local address 127.0.97.62:34673)
0619 13:08:15.830499 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:15.830506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:15.830810 (+ 304us) server_negotiation.cc:407] Connection header received
0619 13:08:15.831015 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:15.831021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:15.831088 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:15.831201 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:15.832037 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.832843 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.834112 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:15.834558 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:15.834692 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:15.838467 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47239: BlockingRecv error: recv got EOF from 127.0.97.2:47239 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:16.282210 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:16.273556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:53691)
0619 13:08:16.274124 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:16.274151 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:16.274269 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:16.274600 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:16.274603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:16.274968 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:16.275114 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.275121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.276112 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.276116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.277267 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.277303 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.278275 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.278279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.278402 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.279115 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:16.279134 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:16.281987 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":272,"threads_started":1}
W20260619 13:08:16.282454 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:16.273714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53691 (local address 127.0.97.62:34673)
0619 13:08:16.273877 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:16.273881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:16.274224 (+ 343us) server_negotiation.cc:407] Connection header received
0619 13:08:16.274383 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:16.274387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:16.274448 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:16.274562 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:16.275234 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.275969 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.277462 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.278141 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.278239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.282215 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53691: BlockingRecv error: recv got EOF from 127.0.97.3:53691 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:16.366077 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:16.356877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:42213)
0619 13:08:16.357429 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:16.357454 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:16.357563 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:16.357893 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:16.357896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:16.358320 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:16.358540 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.358548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.359625 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.359629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.360597 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.360609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.361318 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.361322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.361464 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.362344 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:16.362367 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:16.365766 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":309,"threads_started":1}
W20260619 13:08:16.366323 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:16.356972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42213 (local address 127.0.97.62:34673)
0619 13:08:16.357144 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:16.357148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:16.357548 (+ 400us) server_negotiation.cc:407] Connection header received
0619 13:08:16.357691 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:16.357695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:16.357770 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:16.357849 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:16.358721 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.359480 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.360730 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.361181 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.361281 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.366059 (+ 4778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42213: BlockingRecv error: recv got EOF from 127.0.97.1:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:08:16 dist-test-slave-q11v krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:08:16.848565 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:16.839897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:49035)
0619 13:08:16.840427 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:16.840447 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:16.840563 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:16.841040 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:16.841044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:16.841459 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:16.841680 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.841690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.842712 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.842717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.843698 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:16.843709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.844425 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.844429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:16.844573 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.845164 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:16.845181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:16.848292 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34673: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":285,"threads_started":1}
W20260619 13:08:16.848747 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:16.839959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49035 (local address 127.0.97.62:34673)
0619 13:08:16.840132 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:16.840139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:16.840508 (+ 369us) server_negotiation.cc:407] Connection header received
0619 13:08:16.840801 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:16.840805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:16.840866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:16.840969 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:16.841817 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.842560 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.843820 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:16.844283 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:16.844381 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:16.848484 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49035: BlockingRecv error: recv got EOF from 127.0.97.2:49035 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 13:08:17.106050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9246
I20260619 13:08:17.118263 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:17.107216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.97.1:33485)
0619 13:08:17.107805 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:17.107813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:17.107841 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:08:17.107910 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:17.107917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:17.108145 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:17.108299 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:17.109718 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.110741 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.111917 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.112480 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.112577 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:17.115084 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:17.115093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:17.115109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:17.115141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:17.117776 (+ 2635us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:08:17.118039 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32808: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":327,"threads_started":1}
W20260619 13:08:17.118391 9706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32808: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:08:17.118503 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9246: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33485: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:08:17.118738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9246
I20260619 13:08:17.136605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9397
I20260619 13:08:17.148722 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:17.137743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.97.2:42463)
0619 13:08:17.138413 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:17.138422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:17.138454 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:08:17.138520 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:17.138527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:17.138742 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:17.138926 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:17.140183 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.141236 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.142508 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.143060 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.143163 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:17.145490 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:17.145498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:17.145512 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:17.145547 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:17.148180 (+ 2633us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:08:17.148433 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49766: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":362,"threads_started":1}
W20260619 13:08:17.148833 9707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49766: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:08:17.148883 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9397: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42463: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:08:17.149039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9397
I20260619 13:08:17.166471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9528
I20260619 13:08:17.178519 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:17.167493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.97.3:45147)
0619 13:08:17.168138 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:17.168145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:17.168172 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:08:17.168233 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:17.168240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:17.168477 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:17.168706 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:17.170033 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.171077 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.172204 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.172744 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.172842 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:17.175421 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:17.175432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:17.175445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:17.175479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:17.177956 (+ 2477us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:08:17.178226 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43968: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":319,"threads_started":1}
W20260619 13:08:17.178622 9708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43968: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:08:17.178683 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9528: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45147: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:08:17.178882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9528
I20260619 13:08:17.195261 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9177
I20260619 13:08:17.196635 9239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:17.311244 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9177
2026-06-19T13:08:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21078 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T13:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:08:17Z Disabled control of system clock
I20260619 13:08:17.388012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39653
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:08:17.717269 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:17.717558 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:17.717728 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:17.724998 9713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:08:17.725107 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:17.725178 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:17.725275 9713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:08:17.725355 9713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:08:17.734575 9713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:17.735886 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:17.737746 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:17.749873 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:17.749825 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:17.749919 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:17.750258 9713 server_base.cc:1034] running on GCE node
I20260619 13:08:17.751256 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:17.752732 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:17.753948 9713 hybrid_clock.cc:648] HybridClock initialized: now 1781874497753889 us; error 53 us; skew 500 ppm
I20260619 13:08:17.754369 9713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:17.756762 9713 webserver.cc:460] Webserver started at http://127.0.97.62:39853/ using document root <none> and password file <none>
I20260619 13:08:17.757529 9713 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:17.757670 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:17.757982 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:17.760841 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "90aaa14bf9d8407aae0a02f37830a487"
format_stamp: "Formatted at 2026-06-19 13:08:17 on dist-test-slave-q11v"
server_key: "1c2bfc43b351b6a254d5dd984bd78b12"
server_key_iv: "2e2f5afda72fba7a6d767f7429135805"
server_key_version: "encryptionkey@0"
I20260619 13:08:17.761651 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "90aaa14bf9d8407aae0a02f37830a487"
format_stamp: "Formatted at 2026-06-19 13:08:17 on dist-test-slave-q11v"
server_key: "1c2bfc43b351b6a254d5dd984bd78b12"
server_key_iv: "2e2f5afda72fba7a6d767f7429135805"
server_key_version: "encryptionkey@0"
I20260619 13:08:17.768770 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 13:08:17.773685 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:17.775444 9713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:08:17.775621 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "90aaa14bf9d8407aae0a02f37830a487"
format_stamp: "Formatted at 2026-06-19 13:08:17 on dist-test-slave-q11v"
server_key: "1c2bfc43b351b6a254d5dd984bd78b12"
server_key_iv: "2e2f5afda72fba7a6d767f7429135805"
server_key_version: "encryptionkey@0"
I20260619 13:08:17.775885 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:17.818243 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:17.819140 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:17.819447 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:17.837577 9713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39653
I20260619 13:08:17.837639 9778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39653 every 8 connection(s)
I20260619 13:08:17.838973 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:08:17.842777 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:08:17.845261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9713
I20260619 13:08:17.845455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:08:17.846282 388 external_mini_cluster.cc:1428] Setting key 3601d669997b9c887efff7b261fda138
I20260619 13:08:17.855829 9779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487: Bootstrap starting.
I20260619 13:08:17.859830 9779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:17.861032 9779 log.cc:824] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:17.861696 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:17.848463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.0.97.62:39653)
0619 13:08:17.849532 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:17.849544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:17.849584 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:08:17.851327 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:17.851350 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:17.851826 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:17.852164 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:17.852779 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.854640 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.856535 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:17.857287 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:17.857458 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:17.858898 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:17.858907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:17.858922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:17.859498 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:17.859604 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:17.859632 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:17.859754 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:17.860096 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 13:08:17.860475 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":388,"threads_started":1}
I20260619 13:08:17.864025 9779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487: No bootstrap required, opened a new log
I20260619 13:08:17.871414 9779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260619 13:08:17.871829 9779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:17.871940 9779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90aaa14bf9d8407aae0a02f37830a487, State: Initialized, Role: FOLLOWER
I20260619 13:08:17.872510 9779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260619 13:08:17.872748 9779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:08:17.872900 9779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:08:17.873087 9779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:17.875409 9779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260619 13:08:17.875941 9779 leader_election.cc:302] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90aaa14bf9d8407aae0a02f37830a487; no voters:
I20260619 13:08:17.876529 9779 leader_election.cc:288] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:08:17.876683 9784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:08:17.877470 9784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [term 1 LEADER]: Becoming Leader. State: Replica: 90aaa14bf9d8407aae0a02f37830a487, State: Running, Role: LEADER
I20260619 13:08:17.878091 9784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } }
I20260619 13:08:17.878801 9779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:08:17.880752 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90aaa14bf9d8407aae0a02f37830a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } } }
I20260619 13:08:17.881127 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:17.882153 9790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:08:17.881912 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90aaa14bf9d8407aae0a02f37830a487. Latest consensus state: current_term: 1 leader_uuid: "90aaa14bf9d8407aae0a02f37830a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90aaa14bf9d8407aae0a02f37830a487" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39653 } } }
I20260619 13:08:17.883810 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:17.888011 9790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:08:17.898702 9790 catalog_manager.cc:1347] Generated new cluster ID: 21a62ed3cfca496aa3e9d920ee9fba85
I20260619 13:08:17.898852 9790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:08:17.931176 9790 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:08:17.932590 9790 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:08:17.947270 9790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90aaa14bf9d8407aae0a02f37830a487: Generated new TSK 0
I20260619 13:08:17.948400 9790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:08:17.958372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 13:08:18.297560 9803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:18.297878 9803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:18.298022 9803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:18.306124 9803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:18.306238 9803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:18.306406 9803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:08:18.315740 9803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:18.317128 9803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:18.319006 9803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:18.332945 9811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:18.333067 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:18.333582 9808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:18.333889 9803 server_base.cc:1034] running on GCE node
I20260619 13:08:18.334486 9803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:18.335427 9803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:18.336661 9803 hybrid_clock.cc:648] HybridClock initialized: now 1781874498336613 us; error 62 us; skew 500 ppm
I20260619 13:08:18.337029 9803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:18.339437 9803 webserver.cc:460] Webserver started at http://127.0.97.1:36325/ using document root <none> and password file <none>
I20260619 13:08:18.340176 9803 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:18.340310 9803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:18.340646 9803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:18.343602 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "da285b628bdf40aa911ebee95b0bd2bc"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "ea2fb3300d42a14339e444e808da3c29"
server_key_iv: "0ca4c28b4107ea4f3dc02db91aa31541"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.344455 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "da285b628bdf40aa911ebee95b0bd2bc"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "ea2fb3300d42a14339e444e808da3c29"
server_key_iv: "0ca4c28b4107ea4f3dc02db91aa31541"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.349818 9803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:08:18.353188 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.354754 9803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:08:18.354949 9803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "da285b628bdf40aa911ebee95b0bd2bc"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "ea2fb3300d42a14339e444e808da3c29"
server_key_iv: "0ca4c28b4107ea4f3dc02db91aa31541"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.355173 9803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:18.374286 9803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:18.375253 9803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:18.375572 9803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:18.376649 9803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:18.378334 9803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:18.378430 9803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.378602 9803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:18.378715 9803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.412949 9803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45789
I20260619 13:08:18.413012 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45789 every 8 connection(s)
I20260619 13:08:18.414448 9803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:08:18.417924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9803
I20260619 13:08:18.418134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:08:18.418805 388 external_mini_cluster.cc:1428] Setting key c005991a27688b6913ce6ec222f01603
I20260619 13:08:18.421818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:08:18.428613 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:18.417283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43433 (local address 127.0.97.62:39653)
0619 13:08:18.417865 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:18.417874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:18.419318 (+ 1444us) server_negotiation.cc:407] Connection header received
0619 13:08:18.421257 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:18.421264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:18.421357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:18.421482 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:18.422680 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.423939 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.425485 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.426275 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.426430 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:18.427778 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:18.427788 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:18.427796 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:18.427825 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:18.427884 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:18.427905 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:18.428027 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:18.428260 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 13:08:18.428342 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
I20260619 13:08:18.430207 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:18.417978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.1:43433)
0619 13:08:18.419138 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:18.419183 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:18.420963 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:18.421754 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:18.421762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:18.421853 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:18.422502 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:18.422519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.424084 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.424088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:18.425335 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:18.425346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.426487 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.426492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:18.426706 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:18.427440 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:18.427443 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:18.427482 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:18.427561 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:18.428013 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:18.428021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:18.428026 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:08:18.428209 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 13:08:18.428683 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":375,"threads_started":1}
I20260619 13:08:18.431869 9930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260619 13:08:18.432240 9930 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:18.433115 9930 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260619 13:08:18.435386 9744 ts_manager.cc:194] Registered new tserver with Master: da285b628bdf40aa911ebee95b0bd2bc (127.0.97.1:45789)
I20260619 13:08:18.437132 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43433
W20260619 13:08:18.764257 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:18.764634 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:18.764832 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:18.772513 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:18.772640 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:18.772763 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:08:18.782889 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:18.784200 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:18.786217 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:18.800583 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:18.800544 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:18.800506 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:18.801347 9934 server_base.cc:1034] running on GCE node
I20260619 13:08:18.802083 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:18.803117 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:18.804461 9934 hybrid_clock.cc:648] HybridClock initialized: now 1781874498804345 us; error 67 us; skew 500 ppm
I20260619 13:08:18.804845 9934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:18.807302 9934 webserver.cc:460] Webserver started at http://127.0.97.2:43939/ using document root <none> and password file <none>
I20260619 13:08:18.808035 9934 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:18.808168 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:18.808504 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:18.813058 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "5b16d185292f4db1b0ac31be79a1cab2"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "41fa5ac92758dcdbc35ec9b9d2a9c9ab"
server_key_iv: "328541f23a3f76ac59df73646eb4a759"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.813971 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5b16d185292f4db1b0ac31be79a1cab2"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "41fa5ac92758dcdbc35ec9b9d2a9c9ab"
server_key_iv: "328541f23a3f76ac59df73646eb4a759"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.819218 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:08:18.822688 9948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.824143 9934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:08:18.824328 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "5b16d185292f4db1b0ac31be79a1cab2"
format_stamp: "Formatted at 2026-06-19 13:08:18 on dist-test-slave-q11v"
server_key: "41fa5ac92758dcdbc35ec9b9d2a9c9ab"
server_key_iv: "328541f23a3f76ac59df73646eb4a759"
server_key_version: "encryptionkey@0"
I20260619 13:08:18.824558 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:18.838325 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:18.839188 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:18.839484 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:18.840435 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:18.841971 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:18.842094 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.842236 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:18.842324 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:18.875938 9934 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39869
I20260619 13:08:18.875986 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39869 every 8 connection(s)
I20260619 13:08:18.877303 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:08:18.887225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 9934
I20260619 13:08:18.887393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:08:18.887969 388 external_mini_cluster.cc:1428] Setting key 6bd070e30d72f6f1e974e393f883e381
I20260619 13:08:18.890885 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:18.880436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45513 (local address 127.0.97.62:39653)
0619 13:08:18.880654 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:18.880661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:18.882142 (+ 1481us) server_negotiation.cc:407] Connection header received
0619 13:08:18.883871 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:18.883876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:18.883954 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:18.884047 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:18.885122 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.885958 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.887982 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.888484 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.888626 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:18.890178 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:18.890183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:18.890186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:18.890202 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:18.890231 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:18.890242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:18.890315 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:18.890576 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 13:08:18.890649 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:08:18.891145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:08:18.892508 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:18.880943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.2:45513)
0619 13:08:18.881981 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:18.882019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:18.883582 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:18.884259 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:18.884268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:18.884384 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:18.884974 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:18.884990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.886096 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.886100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:18.887819 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:18.887833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:18.888725 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:18.888733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:18.889011 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:18.889866 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:18.889869 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:18.889908 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:18.890009 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:18.890340 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:18.890352 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:18.890361 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:08:18.890569 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 13:08:18.891015 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":287,"threads_started":1}
I20260619 13:08:18.894001 10061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260619 13:08:18.894307 10061 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:18.895203 10061 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260619 13:08:18.896713 9744 ts_manager.cc:194] Registered new tserver with Master: 5b16d185292f4db1b0ac31be79a1cab2 (127.0.97.2:39869)
I20260619 13:08:18.897804 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45513
W20260619 13:08:19.226197 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:19.226567 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:19.226707 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:19.234395 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:19.234510 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:19.234670 10064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:08:19.243896 10064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:19.245289 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:19.247129 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:19.260691 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:19.260713 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:19.261018 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:19.261773 10064 server_base.cc:1034] running on GCE node
I20260619 13:08:19.262396 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:19.263339 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:19.264554 10064 hybrid_clock.cc:648] HybridClock initialized: now 1781874499264496 us; error 48 us; skew 500 ppm
I20260619 13:08:19.264859 10064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:19.267278 10064 webserver.cc:460] Webserver started at http://127.0.97.3:33377/ using document root <none> and password file <none>
I20260619 13:08:19.267970 10064 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:19.268057 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:19.268335 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:19.271265 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "94a018ed9a924099b00a06dfa5707880"
format_stamp: "Formatted at 2026-06-19 13:08:19 on dist-test-slave-q11v"
server_key: "5ba6d59d2ce7cf8e32f6d3fed0c3a474"
server_key_iv: "6a0f191482dc2cf49582dee76bcab159"
server_key_version: "encryptionkey@0"
I20260619 13:08:19.272024 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "94a018ed9a924099b00a06dfa5707880"
format_stamp: "Formatted at 2026-06-19 13:08:19 on dist-test-slave-q11v"
server_key: "5ba6d59d2ce7cf8e32f6d3fed0c3a474"
server_key_iv: "6a0f191482dc2cf49582dee76bcab159"
server_key_version: "encryptionkey@0"
I20260619 13:08:19.277364 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:08:19.281374 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:19.282919 10064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:08:19.283070 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "94a018ed9a924099b00a06dfa5707880"
format_stamp: "Formatted at 2026-06-19 13:08:19 on dist-test-slave-q11v"
server_key: "5ba6d59d2ce7cf8e32f6d3fed0c3a474"
server_key_iv: "6a0f191482dc2cf49582dee76bcab159"
server_key_version: "encryptionkey@0"
I20260619 13:08:19.283288 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:19.302745 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:19.303619 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:19.303857 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:19.304781 10064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:19.306411 10064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:19.306499 10064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:19.306576 10064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:19.306629 10064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:19.340044 10064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45337
I20260619 13:08:19.340094 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45337 every 8 connection(s)
I20260619 13:08:19.341471 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:08:19.346066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10064
I20260619 13:08:19.346253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:08:19.346848 388 external_mini_cluster.cc:1428] Setting key 718cffb706cde5a418dcf9d4fae98e5e
I20260619 13:08:19.355436 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:19.344193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49367 (local address 127.0.97.62:39653)
0619 13:08:19.344409 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:19.344416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:19.346256 (+ 1840us) server_negotiation.cc:407] Connection header received
0619 13:08:19.348475 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:19.348479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:19.348546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:19.348638 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:19.349946 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.350764 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.352511 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.352945 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.353042 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:19.354649 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:19.354653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:19.354657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:19.354677 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:19.354715 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:19.354729 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:19.354812 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:19.355089 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 13:08:19.355167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:08:19.356550 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:19.344811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39653 (local address 127.0.97.3:49367)
0619 13:08:19.345998 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:19.346062 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:19.348169 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:19.348881 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:19.348916 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:19.349043 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:19.349747 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:19.349769 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.350939 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.350943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:19.352355 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:19.352375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.353202 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.353209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:19.353436 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:19.354332 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:19.354334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:19.354378 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:19.354464 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:19.354867 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:19.354875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:19.354882 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:08:19.355047 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 13:08:19.355376 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":297,"threads_started":1}
I20260619 13:08:19.358106 10191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39653
I20260619 13:08:19.358443 10191 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:19.359344 10191 heartbeater.cc:511] Master 127.0.97.62:39653 requested a full tablet report, sending...
I20260619 13:08:19.360930 9744 ts_manager.cc:194] Registered new tserver with Master: 94a018ed9a924099b00a06dfa5707880 (127.0.97.3:45337)
I20260619 13:08:19.361995 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49367
I20260619 13:08:19.363099 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:08:19.369580 9932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:19.367891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.0.97.62:39653)
0619 13:08:19.368085 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:19.368094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:19.368565 (+ 471us) server_negotiation.cc:407] Connection header received
0619 13:08:19.368794 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:19.368798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:19.368871 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:19.368981 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:19.369386 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56474: BlockingRecv error: recv got EOF from 127.0.0.1:56474 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:08:19.370533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9803
I20260619 13:08:19.378726 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:19.371510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.0.97.1:45789)
0619 13:08:19.372065 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:19.372072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:19.372098 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:08:19.372159 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:19.372165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:19.372412 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:19.372597 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:19.373019 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.374093 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.375326 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.375889 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.376030 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:19.377161 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:19.377170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:19.377184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:19.377871 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:19.377923 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:19.377943 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:19.378068 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:19.378223 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:08:19.378375 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260619 13:08:19.380131 9925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:19.462925 9930 heartbeater.cc:503] Master 127.0.97.62:39653 was elected leader, sending a full tablet report...
I20260619 13:08:19.464627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9803
I20260619 13:08:19.481523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9934
I20260619 13:08:19.489818 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:19.482554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.0.97.2:39869)
0619 13:08:19.483025 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:19.483032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:19.483058 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:08:19.483129 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:19.483135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:19.483364 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:19.483499 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:19.483888 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.484930 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.486323 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.486958 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.487113 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:19.488170 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:19.488179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:19.488193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:19.488955 (+ 762us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:19.489003 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:19.489028 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:19.489115 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:19.489277 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 13:08:19.489403 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260619 13:08:19.491178 10056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:19.573532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9934
I20260619 13:08:19.590266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10064
I20260619 13:08:19.598640 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:19.591261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.0.97.3:45337)
0619 13:08:19.591880 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:19.591887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:19.591918 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:08:19.591978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:19.591985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:19.592217 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:19.592396 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:19.592877 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.594002 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.595245 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:19.595828 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:19.595966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:19.597068 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:19.597077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:19.597090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:19.597774 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:19.597821 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:19.597842 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:19.597968 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:19.598124 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 13:08:19.598272 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":314,"threads_started":1}
I20260619 13:08:19.599952 10186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:19.682936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10064
I20260619 13:08:19.699151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 9713
I20260619 13:08:19.700381 9774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:19.802346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 9713
2026-06-19T13:08:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2485 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T13:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:08:19Z Disabled control of system clock
I20260619 13:08:19.872951 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35601
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:08:20.204545 10210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:20.204855 10210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:20.204996 10210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:20.213117 10210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:08:20.213233 10210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:20.213317 10210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:20.213413 10210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:08:20.213477 10210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:08:20.223065 10210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:20.224407 10210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:20.226300 10210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:20.238245 10215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:20.238299 10216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:20.238289 10218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:20.238973 10210 server_base.cc:1034] running on GCE node
I20260619 13:08:20.239801 10210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:20.241266 10210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:20.242502 10210 hybrid_clock.cc:648] HybridClock initialized: now 1781874500242452 us; error 45 us; skew 500 ppm
I20260619 13:08:20.242864 10210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:20.245110 10210 webserver.cc:460] Webserver started at http://127.0.97.62:34707/ using document root <none> and password file <none>
I20260619 13:08:20.245874 10210 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:20.245998 10210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:20.246322 10210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:20.249230 10210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "8651b7c7ac0a4a278768e93ce75bf36d"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "45ea8f8b13c868a762e8094089500c81"
server_key_iv: "00c4a221b3e51c8c74dd0da0c1680a21"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.250042 10210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "8651b7c7ac0a4a278768e93ce75bf36d"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "45ea8f8b13c868a762e8094089500c81"
server_key_iv: "00c4a221b3e51c8c74dd0da0c1680a21"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.255196 10210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:08:20.258570 10224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:20.259959 10210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:08:20.260164 10210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "8651b7c7ac0a4a278768e93ce75bf36d"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "45ea8f8b13c868a762e8094089500c81"
server_key_iv: "00c4a221b3e51c8c74dd0da0c1680a21"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.260387 10210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:20.265985 10210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:20.266719 10210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:20.267007 10210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:20.285041 10210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35601
I20260619 13:08:20.285073 10275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35601 every 8 connection(s)
I20260619 13:08:20.286432 10210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:08:20.290187 10276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:08:20.294797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10210
I20260619 13:08:20.295019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:08:20.295796 388 external_mini_cluster.cc:1428] Setting key 6fc0a5a139e2428d48c2236aa37a26ab
I20260619 13:08:20.302101 10276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d: Bootstrap starting.
I20260619 13:08:20.305107 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:20.298105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.97.62:35601)
0619 13:08:20.299272 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:20.299288 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:20.299344 (+ 56us) server_negotiation.cc:407] Connection header received
0619 13:08:20.301217 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:20.301243 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:20.301791 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:20.302237 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:20.303360 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:20.303365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:20.303379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:20.303401 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:20.303457 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:20.303475 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:20.303728 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:20.303888 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 13:08:20.304156 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":403,"threads_started":1}
I20260619 13:08:20.305928 10276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:20.307080 10276 log.cc:824] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:20.310111 10276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d: No bootstrap required, opened a new log
I20260619 13:08:20.317929 10276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } }
I20260619 13:08:20.318308 10276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:20.318406 10276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8651b7c7ac0a4a278768e93ce75bf36d, State: Initialized, Role: FOLLOWER
I20260619 13:08:20.318992 10276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } }
I20260619 13:08:20.319240 10276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:08:20.319384 10276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:08:20.319585 10276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:20.321972 10276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } }
I20260619 13:08:20.322662 10276 leader_election.cc:302] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8651b7c7ac0a4a278768e93ce75bf36d; no voters:
I20260619 13:08:20.326076 10276 leader_election.cc:288] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:08:20.326197 10281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:08:20.326929 10281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [term 1 LEADER]: Becoming Leader. State: Replica: 8651b7c7ac0a4a278768e93ce75bf36d, State: Running, Role: LEADER
I20260619 13:08:20.327625 10281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } }
I20260619 13:08:20.329792 10281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8651b7c7ac0a4a278768e93ce75bf36d. Latest consensus state: current_term: 1 leader_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } } }
I20260619 13:08:20.330183 10281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:20.330550 10283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651b7c7ac0a4a278768e93ce75bf36d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35601 } } }
I20260619 13:08:20.330849 10283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:20.331714 10276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:08:20.338852 10285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:08:20.344614 10285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:08:20.354890 10285 catalog_manager.cc:1347] Generated new cluster ID: aeb460faf9204cd39d55da02dbce1e86
I20260619 13:08:20.355020 10285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:08:20.379720 10285 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:08:20.380349 10285 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:08:20.393086 10285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8651b7c7ac0a4a278768e93ce75bf36d: Generated new TSK 0
I20260619 13:08:20.394261 10285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:08:20.408711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:08:20.744915 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:20.745283 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:20.745436 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:20.753773 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:20.753930 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:20.754130 10300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:08:20.763693 10300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:20.765161 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:20.767019 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:20.780493 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:20.780582 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:20.780459 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:20.781389 10300 server_base.cc:1034] running on GCE node
I20260619 13:08:20.782092 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:20.783113 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:20.784358 10300 hybrid_clock.cc:648] HybridClock initialized: now 1781874500784284 us; error 59 us; skew 500 ppm
I20260619 13:08:20.784706 10300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:20.787137 10300 webserver.cc:460] Webserver started at http://127.0.97.1:46073/ using document root <none> and password file <none>
I20260619 13:08:20.787844 10300 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:20.787976 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:20.788301 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:20.791282 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a61b51cd6f34d35a7d4998840e6e775"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "db8680b34dad2b9972bcc99a81655f24"
server_key_iv: "da3f18e94974297f1f571866dcb450bc"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.792079 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a61b51cd6f34d35a7d4998840e6e775"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "db8680b34dad2b9972bcc99a81655f24"
server_key_iv: "da3f18e94974297f1f571866dcb450bc"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.797206 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:08:20.800838 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:20.802549 10300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:08:20.802736 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "4a61b51cd6f34d35a7d4998840e6e775"
format_stamp: "Formatted at 2026-06-19 13:08:20 on dist-test-slave-q11v"
server_key: "db8680b34dad2b9972bcc99a81655f24"
server_key_iv: "da3f18e94974297f1f571866dcb450bc"
server_key_version: "encryptionkey@0"
I20260619 13:08:20.802989 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:20.808559 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:20.809451 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:20.809777 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:20.810745 10300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:20.812131 10300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:20.812256 10300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:20.812379 10300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:20.812469 10300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:20.846141 10300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42029
I20260619 13:08:20.846231 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42029 every 8 connection(s)
I20260619 13:08:20.847555 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:08:20.851732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10300
I20260619 13:08:20.851945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:08:20.852584 388 external_mini_cluster.cc:1428] Setting key f1acaa99678701b35896e3b0ab4f750e
I20260619 13:08:20.855456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 13:08:20.856515 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:20.850235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59931 (local address 127.0.97.62:35601)
0619 13:08:20.850655 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:20.850662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:20.852021 (+ 1359us) server_negotiation.cc:407] Connection header received
0619 13:08:20.854024 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:20.854030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:20.854095 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:20.854241 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:20.855759 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:20.855771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:20.855774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:20.855792 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:20.855831 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:20.855842 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:20.855941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:20.856228 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 13:08:20.856283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":193,"threads_started":1}
I20260619 13:08:20.857931 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:20.850928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35601 (local address 127.0.97.1:59931)
0619 13:08:20.851837 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:20.851890 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:20.853756 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:20.854548 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:20.854557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:20.854662 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:20.855379 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:20.855382 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:20.855423 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:20.855513 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:20.855951 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:20.855965 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:20.855978 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:08:20.856281 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 13:08:20.856561 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260619 13:08:20.859622 10427 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35601
I20260619 13:08:20.860028 10427 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:20.860679 10427 heartbeater.cc:511] Master 127.0.97.62:35601 requested a full tablet report, sending...
I20260619 13:08:20.863004 10241 ts_manager.cc:194] Registered new tserver with Master: 4a61b51cd6f34d35a7d4998840e6e775 (127.0.97.1:42029)
W20260619 13:08:21.207996 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:21.208312 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:21.208410 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:21.216099 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:21.216228 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:21.216358 10431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:08:21.225857 10431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:21.227161 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:21.228981 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:21.243543 10436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:21.243572 10437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:21.243808 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:21.244971 10431 server_base.cc:1034] running on GCE node
I20260619 13:08:21.245779 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:21.246829 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:21.248073 10431 hybrid_clock.cc:648] HybridClock initialized: now 1781874501248022 us; error 49 us; skew 500 ppm
I20260619 13:08:21.248409 10431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:21.250777 10431 webserver.cc:460] Webserver started at http://127.0.97.2:42437/ using document root <none> and password file <none>
I20260619 13:08:21.251492 10431 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:21.251585 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:21.251935 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:21.255046 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "8a06755e03e946ce8c25d21e4d48113e"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "3405de44e0263fb84e22493a6055b586"
server_key_iv: "a024fa1d606178494b72a3e4167c9001"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.255861 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8a06755e03e946ce8c25d21e4d48113e"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "3405de44e0263fb84e22493a6055b586"
server_key_iv: "a024fa1d606178494b72a3e4167c9001"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.261471 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:08:21.265698 10445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.267303 10431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:08:21.267503 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "8a06755e03e946ce8c25d21e4d48113e"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "3405de44e0263fb84e22493a6055b586"
server_key_iv: "a024fa1d606178494b72a3e4167c9001"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.267728 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:21.273885 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:21.274727 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:21.275018 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:21.275991 10431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:21.277431 10431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:21.277522 10431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.277751 10431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:21.277849 10431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.312585 10431 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38085
I20260619 13:08:21.312642 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38085 every 8 connection(s)
I20260619 13:08:21.314023 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:08:21.322172 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.316660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42835 (local address 127.0.97.62:35601)
0619 13:08:21.316862 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:21.316868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:21.318371 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 13:08:21.320022 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:21.320027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:21.320082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:21.320174 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:21.321501 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:21.321505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:21.321509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:21.321524 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:21.321555 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:21.321566 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:21.321730 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:21.321910 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 13:08:21.321960 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:08:21.323172 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.317245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35601 (local address 127.0.97.2:42835)
0619 13:08:21.318198 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:21.318243 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:21.319721 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:21.320383 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:21.320391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:21.320484 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:21.321197 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:21.321199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:21.321237 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:21.321323 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:21.321707 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:21.321715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:21.321720 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:08:21.321957 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 13:08:21.322179 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":326,"threads_started":1}
I20260619 13:08:21.323284 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10431
I20260619 13:08:21.323494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:08:21.324071 388 external_mini_cluster.cc:1428] Setting key 1e2ff46eca0c1592640863104a7f9fac
I20260619 13:08:21.324807 10558 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35601
I20260619 13:08:21.325127 10558 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:21.325766 10558 heartbeater.cc:511] Master 127.0.97.62:35601 requested a full tablet report, sending...
I20260619 13:08:21.327090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 13:08:21.327315 10241 ts_manager.cc:194] Registered new tserver with Master: 8a06755e03e946ce8c25d21e4d48113e (127.0.97.2:38085)
W20260619 13:08:21.662081 10561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:21.662380 10561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:21.662469 10561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:21.670058 10561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:21.670171 10561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:21.670281 10561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:08:21.679805 10561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:21.681072 10561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:21.682842 10561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:21.696412 10566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:21.696521 10567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:21.696511 10569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:21.697417 10561 server_base.cc:1034] running on GCE node
I20260619 13:08:21.698163 10561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:21.699119 10561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:21.700407 10561 hybrid_clock.cc:648] HybridClock initialized: now 1781874501700349 us; error 50 us; skew 500 ppm
I20260619 13:08:21.700873 10561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:08:21.703411 10561 webserver.cc:460] Webserver started at http://127.0.97.3:40425/ using document root <none> and password file <none>
I20260619 13:08:21.704171 10561 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:21.704320 10561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:21.704665 10561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:21.707793 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "1e28d6d83b624059b2a6e66c02a61e42"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "f16764d7af65b0b3caa63e246263e14c"
server_key_iv: "a714f2780fa5e60e46a5945c1f3435b2"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.708657 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1e28d6d83b624059b2a6e66c02a61e42"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "f16764d7af65b0b3caa63e246263e14c"
server_key_iv: "a714f2780fa5e60e46a5945c1f3435b2"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.714416 10561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:08:21.718109 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.719508 10561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:08:21.719729 10561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "1e28d6d83b624059b2a6e66c02a61e42"
format_stamp: "Formatted at 2026-06-19 13:08:21 on dist-test-slave-q11v"
server_key: "f16764d7af65b0b3caa63e246263e14c"
server_key_iv: "a714f2780fa5e60e46a5945c1f3435b2"
server_key_version: "encryptionkey@0"
I20260619 13:08:21.720042 10561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:21.725914 10561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:21.726713 10561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:21.726990 10561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:21.727880 10561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:21.729239 10561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:21.729362 10561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.729497 10561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:21.729588 10561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:21.764441 10561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46487
I20260619 13:08:21.764472 10687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46487 every 8 connection(s)
I20260619 13:08:21.765955 10561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:08:21.771018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10561
I20260619 13:08:21.771214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:08:21.771790 388 external_mini_cluster.cc:1428] Setting key db4d4efd854f9a99e08c140e4849cb66
I20260619 13:08:21.775323 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.768648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38129 (local address 127.0.97.62:35601)
0619 13:08:21.768830 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:21.768835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:21.770382 (+ 1547us) server_negotiation.cc:407] Connection header received
0619 13:08:21.772537 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:21.772543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:21.772611 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:21.772730 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:21.774514 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:21.774519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:21.774523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:21.774541 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:21.774572 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:21.774584 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:21.774730 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:21.774941 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 13:08:21.775012 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:08:21.776324 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.769212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35601 (local address 127.0.97.3:38129)
0619 13:08:21.770225 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:21.770268 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:21.772170 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:21.773041 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:21.773052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:21.773198 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:21.774152 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:08:21.774158 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:21.774211 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:08:21.774331 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:21.774696 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:21.774707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:21.774716 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:08:21.775026 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 13:08:21.775308 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
I20260619 13:08:21.777834 10688 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35601
I20260619 13:08:21.778228 10688 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:21.778815 10688 heartbeater.cc:511] Master 127.0.97.62:35601 requested a full tablet report, sending...
I20260619 13:08:21.780285 10241 ts_manager.cc:194] Registered new tserver with Master: 1e28d6d83b624059b2a6e66c02a61e42 (127.0.97.3:46487)
I20260619 13:08:21.787487 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:08:21.793804 10429 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:08:21.792048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.0.97.62:35601)
0619 13:08:21.792219 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:21.792224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:21.792746 (+ 522us) server_negotiation.cc:407] Connection header received
0619 13:08:21.792975 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:21.792980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:21.793031 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:21.793139 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:21.793481 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45552: BlockingRecv error: recv got EOF from 127.0.0.1:45552 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:08:21.794615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10300
I20260619 13:08:21.798473 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.795579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.97.1:42029)
0619 13:08:21.796085 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:21.796092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:21.796117 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:08:21.796193 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:21.796199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:21.796427 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:21.796604 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:21.797705 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:21.797712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:21.797724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:21.797736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:21.797779 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:21.797792 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:21.797933 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:21.798056 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:08:21.798158 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20260619 13:08:21.800031 10422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:21.885047 10427 heartbeater.cc:503] Master 127.0.97.62:35601 was elected leader, sending a full tablet report...
I20260619 13:08:21.886515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10300
W20260619 13:08:21.903333 10226 connection.cc:535] server connection from 127.0.97.1:59931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:08:21.903637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10431
I20260619 13:08:21.907667 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:21.904761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.0.97.2:38085)
0619 13:08:21.905346 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:21.905353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:21.905380 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:08:21.905441 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:21.905446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:21.905751 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:21.905880 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:21.906901 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:21.906908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:21.906921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:21.906939 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:21.906983 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:21.906997 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:21.907131 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:21.907347 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 13:08:21.907429 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
I20260619 13:08:21.908939 10553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:21.991137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10431
I20260619 13:08:22.007381 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10561
I20260619 13:08:22.011505 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:22.008503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.0.97.3:46487)
0619 13:08:22.009157 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:22.009165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:22.009197 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:08:22.009261 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:22.009268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:22.009502 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:22.009743 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:22.010662 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:22.010669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:22.010682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:08:22.010697 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:22.010738 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:08:22.010752 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:22.010890 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:22.011035 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 13:08:22.011120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":337,"threads_started":1}
I20260619 13:08:22.012928 10683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:22.095230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10561
I20260619 13:08:22.111364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10210
I20260619 13:08:22.112727 10271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:22.220026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10210
2026-06-19T13:08:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2419 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:08:22 dist-test-slave-q11v krb5kdc[10706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:08:22 dist-test-slave-q11v krb5kdc[10706](info): set up 2 sockets
Jun 19 13:08:22 dist-test-slave-q11v krb5kdc[10706](info): commencing operation
krb5kdc: starting...
W20260619 13:08:23.331079 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:08:23 dist-test-slave-q11v krb5kdc[10706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:08:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:08:23.573472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40431
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:08:23.902691 10719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:23.903000 10719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:23.903165 10719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:23.910450 10719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:08:23.910565 10719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:23.910661 10719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:23.910764 10719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:08:23.910851 10719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:08:23.920821 10719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:23.922200 10719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:23.923545 10719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:23.935407 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:23.935415 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:23.935451 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:23.936307 10719 server_base.cc:1034] running on GCE node
I20260619 13:08:23.937130 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:23.938733 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:23.939970 10719 hybrid_clock.cc:648] HybridClock initialized: now 1781874503939924 us; error 40 us; skew 500 ppm
I20260619 13:08:23.940313 10719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:23 dist-test-slave-q11v krb5kdc[10706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:23.947187 10719 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:08:23.948762 10719 webserver.cc:460] Webserver started at http://127.0.97.62:42951/ using document root <none> and password file <none>
I20260619 13:08:23.949483 10719 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:23.949684 10719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:23.950007 10719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:23.953071 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "856dfac99d89449d8fb395d662d21140"
format_stamp: "Formatted at 2026-06-19 13:08:23 on dist-test-slave-q11v"
server_key: "4ab5847216182da29df9cb11a867b185"
server_key_iv: "b3ba527f339a5a084008206375f2613b"
server_key_version: "encryptionkey@0"
I20260619 13:08:23.953933 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "856dfac99d89449d8fb395d662d21140"
format_stamp: "Formatted at 2026-06-19 13:08:23 on dist-test-slave-q11v"
server_key: "4ab5847216182da29df9cb11a867b185"
server_key_iv: "b3ba527f339a5a084008206375f2613b"
server_key_version: "encryptionkey@0"
I20260619 13:08:23.959164 10719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:08:23.962620 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:23.964057 10719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:08:23.964247 10719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "856dfac99d89449d8fb395d662d21140"
format_stamp: "Formatted at 2026-06-19 13:08:23 on dist-test-slave-q11v"
server_key: "4ab5847216182da29df9cb11a867b185"
server_key_iv: "b3ba527f339a5a084008206375f2613b"
server_key_version: "encryptionkey@0"
I20260619 13:08:23.964452 10719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:23.990377 10719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:23.994545 10719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:23.994841 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:24.014134 10719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40431
I20260619 13:08:24.014166 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40431 every 8 connection(s)
I20260619 13:08:24.015596 10719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:08:24.016279 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10719
I20260619 13:08:24.016573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:08:24.017305 388 external_mini_cluster.cc:1428] Setting key 609fae583c320788b7d3e13b824d9baf
I20260619 13:08:24.020345 10786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:08:24 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:08:24.033043 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: Bootstrap starting.
I20260619 13:08:24.037406 10786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:24.038694 10786 log.cc:824] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:24.042196 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: No bootstrap required, opened a new log
I20260619 13:08:24.042333 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:24.019451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.0.97.62:40431)
0619 13:08:24.020231 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:24.020247 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:24.020314 (+ 67us) server_negotiation.cc:407] Connection header received
0619 13:08:24.022646 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:24.022674 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:24.023153 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:24.023662 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:24.024901 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.027014 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.028365 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.029394 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.029584 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:24.033522 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:24.033532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:24.033553 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:24.033621 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:24.037702 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:24.038450 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:24.038462 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:24.038478 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:24.038713 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:24.039134 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:24.039140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:24.039143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:24.039816 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:24.040027 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:24.040552 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 13:08:24.040974 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":321,"threads_started":1}
I20260619 13:08:24.050732 10786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } }
I20260619 13:08:24.051122 10786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:24.051194 10786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856dfac99d89449d8fb395d662d21140, State: Initialized, Role: FOLLOWER
I20260619 13:08:24.051694 10786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } }
I20260619 13:08:24.051905 10786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:08:24.051992 10786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:08:24.052119 10786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:24.054425 10786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } }
I20260619 13:08:24.054908 10786 leader_election.cc:302] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856dfac99d89449d8fb395d662d21140; no voters:
I20260619 13:08:24.055500 10786 leader_election.cc:288] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:08:24.055661 10791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:08:24.056370 10791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [term 1 LEADER]: Becoming Leader. State: Replica: 856dfac99d89449d8fb395d662d21140, State: Running, Role: LEADER
I20260619 13:08:24.056900 10791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } }
I20260619 13:08:24.057771 10786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:08:24.059566 10792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "856dfac99d89449d8fb395d662d21140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } } }
I20260619 13:08:24.059907 10792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:24.060047 10793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 856dfac99d89449d8fb395d662d21140. Latest consensus state: current_term: 1 leader_uuid: "856dfac99d89449d8fb395d662d21140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856dfac99d89449d8fb395d662d21140" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40431 } } }
I20260619 13:08:24.060609 10793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140 [sys.catalog]: This master's current role is: LEADER
I20260619 13:08:24.062616 10798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:08:24.069897 10798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:08:24.080149 10798 catalog_manager.cc:1347] Generated new cluster ID: 17d8000a1cc24a2184b0f18575363255
I20260619 13:08:24.080327 10798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:08:24.089018 10798 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:08:24.090539 10798 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:08:24.099573 10798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: Generated new TSK 0
I20260619 13:08:24.100492 10798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:08:24.174362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:08:24.531708 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:24.532027 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:24.532122 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:24.539471 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:24.539593 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:24.539706 10812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:08:24.549755 10812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:24.551153 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:24.552778 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:24.566958 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:24.567001 10818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:24.567520 10820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:24.568118 10812 server_base.cc:1034] running on GCE node
I20260619 13:08:24.568720 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:24.569904 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:24.571121 10812 hybrid_clock.cc:648] HybridClock initialized: now 1781874504571080 us; error 28 us; skew 500 ppm
I20260619 13:08:24.571517 10812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:24 dist-test-slave-q11v krb5kdc[10706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:24.574714 10812 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:08:24.576335 10812 webserver.cc:460] Webserver started at http://127.0.97.1:34327/ using document root <none> and password file <none>
I20260619 13:08:24.577071 10812 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:24.577208 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:24.577539 10812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:24.580514 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "4bb81076d591497d9299f486e71688aa"
format_stamp: "Formatted at 2026-06-19 13:08:24 on dist-test-slave-q11v"
server_key: "c276fe7f6ae2e5fec4985d0d3c7140c3"
server_key_iv: "3039443c0f6ba4a46d7413774aae936b"
server_key_version: "encryptionkey@0"
I20260619 13:08:24.581298 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4bb81076d591497d9299f486e71688aa"
format_stamp: "Formatted at 2026-06-19 13:08:24 on dist-test-slave-q11v"
server_key: "c276fe7f6ae2e5fec4985d0d3c7140c3"
server_key_iv: "3039443c0f6ba4a46d7413774aae936b"
server_key_version: "encryptionkey@0"
I20260619 13:08:24.586651 10812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 13:08:24.590387 10827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:24.591950 10812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:08:24.592128 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "4bb81076d591497d9299f486e71688aa"
format_stamp: "Formatted at 2026-06-19 13:08:24 on dist-test-slave-q11v"
server_key: "c276fe7f6ae2e5fec4985d0d3c7140c3"
server_key_iv: "3039443c0f6ba4a46d7413774aae936b"
server_key_version: "encryptionkey@0"
I20260619 13:08:24.592356 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:24.609148 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:24.613510 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:24.613869 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:24.614827 10812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:24.616178 10812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:24.616253 10812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:24.616386 10812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:24.616459 10812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:24.650543 10812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36845
I20260619 13:08:24.650564 10939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36845 every 8 connection(s)
I20260619 13:08:24.651965 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:08:24.653095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10812
I20260619 13:08:24.653328 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:08:24.654028 388 external_mini_cluster.cc:1428] Setting key e85cd45540c8cfd4eeb27727165b6ae9
Jun 19 13:08:24 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:08:24.674264 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:24.655009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49667 (local address 127.0.97.62:40431)
0619 13:08:24.655435 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:24.655441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:24.656663 (+ 1222us) server_negotiation.cc:407] Connection header received
0619 13:08:24.658688 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:24.658696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:24.658784 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:24.658898 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:24.660433 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.661633 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.663932 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.664415 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.664564 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:24.668215 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:24.668221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:24.668235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:24.668269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:24.670977 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:24.671738 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:24.671744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:24.671748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:24.671870 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:24.672310 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:24.672315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:24.672318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:24.672602 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:24.672780 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:24.673829 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0619 13:08:24.674007 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":195,"threads_started":1}
I20260619 13:08:24.676090 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:24.655632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40431 (local address 127.0.97.1:49667)
0619 13:08:24.656495 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:24.656543 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:24.658414 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:24.659133 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:24.659142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:24.659656 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:24.660253 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:24.660271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.661805 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.661813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:24.663756 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:24.663769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:24.664646 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:24.664651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:24.664880 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:24.665688 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:24.665720 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:24.667966 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:24.671138 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:24.671147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:24.671166 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:24.671606 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:24.671983 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:24.671988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:24.671991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:24.672188 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:24.672789 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:24.672801 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:24.673335 (+ 534us) client_negotiation.cc:769] Sending connection context
0619 13:08:24.673758 (+ 423us) client_negotiation.cc:240] Negotiation successful
0619 13:08:24.674471 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":383,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:08:24.677878 10940 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40431
I20260619 13:08:24.678196 10940 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:24.679025 10940 heartbeater.cc:511] Master 127.0.97.62:40431 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:08:24.681214 10751 ts_manager.cc:194] Registered new tserver with Master: 4bb81076d591497d9299f486e71688aa (127.0.97.1:36845)
I20260619 13:08:24.682741 10751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:08:24.706550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:08:25.042820 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:25.043210 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:25.043413 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:25.051038 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:25.051211 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:25.051400 10946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:08:25.061833 10946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:25.063318 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:25.064714 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:25.078830 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:25.078868 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:25.078895 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:25.079825 10946 server_base.cc:1034] running on GCE node
I20260619 13:08:25.080533 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:25.081653 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:25.082886 10946 hybrid_clock.cc:648] HybridClock initialized: now 1781874505082840 us; error 44 us; skew 500 ppm
I20260619 13:08:25.083226 10946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:25 dist-test-slave-q11v krb5kdc[10706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:25.086627 10946 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:08:25.088164 10946 webserver.cc:460] Webserver started at http://127.0.97.2:38669/ using document root <none> and password file <none>
I20260619 13:08:25.088928 10946 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:25.089077 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:25.089411 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:25.092576 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "dcac482cc3db41d99395219dcd0bf95d"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "bf058bcd2eee54e67bf31b0d005a6ea8"
server_key_iv: "cded60df76578fc2dd30c837b6b89270"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.093405 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dcac482cc3db41d99395219dcd0bf95d"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "bf058bcd2eee54e67bf31b0d005a6ea8"
server_key_iv: "cded60df76578fc2dd30c837b6b89270"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.099050 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:08:25.102753 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.104194 10946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:08:25.104400 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "dcac482cc3db41d99395219dcd0bf95d"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "bf058bcd2eee54e67bf31b0d005a6ea8"
server_key_iv: "cded60df76578fc2dd30c837b6b89270"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.104616 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:25.127377 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:25.131970 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:25.132273 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:25.133311 10946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:25.134750 10946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:25.134860 10946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.135012 10946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:25.135085 10946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.176116 10946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33691
I20260619 13:08:25.176157 11073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33691 every 8 connection(s)
I20260619 13:08:25.177563 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:08:25.182317 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 10946
I20260619 13:08:25.182523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:08:25.182950 388 external_mini_cluster.cc:1428] Setting key 952fa1e704c47ecc51d931272a704482
Jun 19 13:08:25 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:08:25.198794 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.180151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34375 (local address 127.0.97.62:40431)
0619 13:08:25.180614 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.180618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.181783 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 13:08:25.184019 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.184025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.184119 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.184240 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:25.185997 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.186737 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.188287 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.188738 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.188872 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.192660 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:25.192666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:25.192670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:25.192703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:25.195407 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.196255 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.196264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.196267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.196392 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.196902 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.196912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.196914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.197217 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:25.197371 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.198171 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.198394 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":253,"threads_started":1}
I20260619 13:08:25.199833 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.180789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40431 (local address 127.0.97.2:34375)
0619 13:08:25.181536 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:25.181577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:25.183757 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:25.184486 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:25.184495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:25.185143 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:25.185828 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.185848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.186898 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.186903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.188126 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.188138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.188973 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.188980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.189204 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.189958 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:25.189982 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:25.192410 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:25.195564 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:25.195574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:25.195592 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:25.196137 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:25.196574 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:25.196579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:25.196583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:25.196775 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:25.197347 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:25.197355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:25.197802 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 13:08:25.198081 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 13:08:25.198657 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":274,"threads_started":1}
I20260619 13:08:25.201377 11074 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40431
I20260619 13:08:25.201761 11074 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:25.202486 11074 heartbeater.cc:511] Master 127.0.97.62:40431 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:08:25.204077 10751 ts_manager.cc:194] Registered new tserver with Master: dcac482cc3db41d99395219dcd0bf95d (127.0.97.2:33691)
I20260619 13:08:25.205019 10751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:08:25.233260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:08:25.565949 11080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:08:25.566241 11080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:08:25.566404 11080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:08:25.573868 11080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:08:25.573987 11080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:08:25.574168 11080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:08:25.584038 11080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:08:25.585402 11080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:08:25.586781 11080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:08:25.600664 11085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:25.600623 11086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:08:25.600736 11088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:08:25.601395 11080 server_base.cc:1034] running on GCE node
I20260619 13:08:25.602075 11080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:08:25.603071 11080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:08:25.604317 11080 hybrid_clock.cc:648] HybridClock initialized: now 1781874505604266 us; error 47 us; skew 500 ppm
I20260619 13:08:25.604653 11080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:08:25 dist-test-slave-q11v krb5kdc[10706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:08:25.607935 11080 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:08:25.609653 11080 webserver.cc:460] Webserver started at http://127.0.97.3:42287/ using document root <none> and password file <none>
I20260619 13:08:25.610364 11080 fs_manager.cc:359] Metadata directory not provided
I20260619 13:08:25.610466 11080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:08:25.610698 11080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:08:25.613766 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "701e17423ac8498086eaeee9148c482f"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "4a594a283ac394cb8bec703d39bfabce"
server_key_iv: "8cc2a4fc37d0f3d9d091dde55bd33919"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.614610 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "701e17423ac8498086eaeee9148c482f"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "4a594a283ac394cb8bec703d39bfabce"
server_key_iv: "8cc2a4fc37d0f3d9d091dde55bd33919"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.619985 11080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 13:08:25.623660 11095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.625152 11080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:08:25.625283 11080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "701e17423ac8498086eaeee9148c482f"
format_stamp: "Formatted at 2026-06-19 13:08:25 on dist-test-slave-q11v"
server_key: "4a594a283ac394cb8bec703d39bfabce"
server_key_iv: "8cc2a4fc37d0f3d9d091dde55bd33919"
server_key_version: "encryptionkey@0"
I20260619 13:08:25.625454 11080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:08:25.651988 11080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:08:25.656739 11080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:08:25.657064 11080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:08:25.658221 11080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:08:25.659737 11080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:08:25.659819 11080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.659998 11080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:08:25.660049 11080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:08:25.686620 10940 heartbeater.cc:503] Master 127.0.97.62:40431 was elected leader, sending a full tablet report...
I20260619 13:08:25.693270 11080 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33815
I20260619 13:08:25.693313 11207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33815 every 8 connection(s)
I20260619 13:08:25.694653 11080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:08:25.696640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 11080
I20260619 13:08:25.696852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:08:25.697361 388 external_mini_cluster.cc:1428] Setting key 6073600210e9bee1a1c65a17139581e4
Jun 19 13:08:25 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:08:25.714745 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.697401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41721 (local address 127.0.97.62:40431)
0619 13:08:25.697636 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.697642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.699026 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 13:08:25.700856 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.700860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.700986 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.701108 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:25.702626 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.703344 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.704803 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.705212 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.705307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.709008 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:25.709013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:25.709016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:25.709048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:25.711651 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.712430 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.712435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.712445 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.712559 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.712975 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.712980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.712984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.713265 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:25.713413 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.714191 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.714419 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 13:08:25.715700 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.697994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40431 (local address 127.0.97.3:41721)
0619 13:08:25.698848 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:25.698902 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:25.700582 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:25.701311 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:25.701319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:25.701895 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:08:25.702473 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.702491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.703468 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.703472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.704670 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.704681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.705400 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.705404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.705652 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.706550 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:08:25.706571 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:08:25.708827 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:08:25.711810 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:25.711818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:25.711833 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:25.712289 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:25.712668 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:08:25.712687 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:08:25.712690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:08:25.712856 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:08:25.713400 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:08:25.713408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:08:25.713815 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 13:08:25.714082 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 13:08:25.714594 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":368,"threads_started":1}
I20260619 13:08:25.716979 11208 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40431
I20260619 13:08:25.717288 11208 heartbeater.cc:464] Registering TS with master...
I20260619 13:08:25.718050 11208 heartbeater.cc:511] Master 127.0.97.62:40431 requested a full tablet report, sending...
I20260619 13:08:25.719506 10751 ts_manager.cc:194] Registered new tserver with Master: 701e17423ac8498086eaeee9148c482f (127.0.97.3:33815)
I20260619 13:08:25.720381 10751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41721
I20260619 13:08:25.725061 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:08:25.742267 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.730036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.0.97.62:40431)
0619 13:08:25.730286 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.730296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.730744 (+ 448us) server_negotiation.cc:407] Connection header received
0619 13:08:25.730889 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.730894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.730959 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.731050 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:25.732374 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.733308 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.734573 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.734963 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.735067 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.737191 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:25.737196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:25.737199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:25.737232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:25.739910 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.740487 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.740491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.740495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.740618 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:25.740926 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:25.740929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:25.740931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:25.741244 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:25.741428 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.741741 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.741894 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:08:25.746503 10751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:08:25.753811 10751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:08:25.779192 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.768510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.0.97.2:33691)
0619 13:08:25.770026 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.770037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.770066 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:08:25.770165 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.770174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.770488 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.771581 (+ 1093us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:08:25.771852 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.773457 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.776295 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.777401 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.777559 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.777828 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.778679 (+ 851us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.778883 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1091,"threads_started":1}
I20260619 13:08:25.779206 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.767892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33691 (local address 127.0.0.1:59614)
0619 13:08:25.769178 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:25.769200 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:25.769343 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:25.770737 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:25.770744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:25.770774 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:08:25.771358 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.771373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.773677 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.773683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.776118 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.776131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.777677 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.777685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.777918 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.777983 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 13:08:25.778121 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 13:08:25.778319 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":112,"thread_start_us":952,"threads_started":1}
I20260619 13:08:25.780678 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.768103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.97.3:33815)
0619 13:08:25.770331 (+ 2228us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.770339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.770371 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:08:25.770472 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.770479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.771769 (+ 1290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.771982 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:08:25.772754 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.774285 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.776675 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.778178 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.778297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.778521 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.780174 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.780332 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063,"thread_start_us":1852,"threads_started":1}
I20260619 13:08:25.780983 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.770791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36845 (local address 127.0.0.1:45082)
0619 13:08:25.771465 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:25.771484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:25.771580 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:25.773253 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:25.773258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:25.773284 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:08:25.773669 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.773682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.775655 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.775660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.777787 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.777801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.779925 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.779931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.780083 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.780124 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 13:08:25.780217 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 13:08:25.780355 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":5,"thread_start_us":336,"threads_started":1}
I20260619 13:08:25.781141 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.767556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33815 (local address 127.0.0.1:60344)
0619 13:08:25.768443 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:25.768497 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:25.768675 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:25.771998 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:25.772006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:25.772039 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:08:25.772503 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.772515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.774503 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.774508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.776506 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:25.776520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.778688 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.778694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:25.778883 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.778919 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:08:25.780119 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0619 13:08:25.780254 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":46,"thread_start_us":423,"threads_started":1}
I20260619 13:08:25.781651 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:25.771151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.0.97.1:36845)
0619 13:08:25.772680 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:25.772688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:25.772713 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:08:25.772805 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:25.772812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:25.773046 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:25.773230 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:08:25.773883 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.775505 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.778186 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:25.779735 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:25.779860 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:25.780010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:25.780484 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 13:08:25.780648 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1365,"thread_start_us":1224,"threads_started":1}
I20260619 13:08:25.783407 11009 tablet_service.cc:1467] Processing CreateTablet for tablet 06f70b420e0c4280811e0b51c47e096e (DEFAULT_TABLE table=test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:08:25.784849 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f70b420e0c4280811e0b51c47e096e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:08:25.784934 11143 tablet_service.cc:1467] Processing CreateTablet for tablet 06f70b420e0c4280811e0b51c47e096e (DEFAULT_TABLE table=test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:08:25.784934 10875 tablet_service.cc:1467] Processing CreateTablet for tablet 06f70b420e0c4280811e0b51c47e096e (DEFAULT_TABLE table=test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:08:25.786361 10875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f70b420e0c4280811e0b51c47e096e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:08:25.786352 11143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06f70b420e0c4280811e0b51c47e096e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:08:25.798138 11226 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Bootstrap starting.
I20260619 13:08:25.800982 11228 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Bootstrap starting.
I20260619 13:08:25.801223 11226 tablet_bootstrap.cc:654] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:25.801919 11227 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Bootstrap starting.
I20260619 13:08:25.802577 11226 log.cc:824] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:25.804979 11228 tablet_bootstrap.cc:654] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:25.805230 11227 tablet_bootstrap.cc:654] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Neither blocks nor log segments found. Creating new log.
I20260619 13:08:25.806267 11228 log.cc:824] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:25.806433 11227 log.cc:824] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:08:25.806706 11226 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: No bootstrap required, opened a new log
I20260619 13:08:25.807018 11226 ts_tablet_manager.cc:1397] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:08:25.809497 11227 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: No bootstrap required, opened a new log
I20260619 13:08:25.809633 11228 tablet_bootstrap.cc:492] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: No bootstrap required, opened a new log
I20260619 13:08:25.809850 11227 ts_tablet_manager.cc:1397] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 13:08:25.809901 11228 ts_tablet_manager.cc:1397] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:08:25.816859 11227 raft_consensus.cc:348] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.817260 11227 raft_consensus.cc:374] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:25.817353 11227 raft_consensus.cc:729] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701e17423ac8498086eaeee9148c482f, State: Initialized, Role: FOLLOWER
I20260619 13:08:25.818076 11227 consensus_queue.cc:260] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.818174 11226 raft_consensus.cc:348] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.818638 11226 raft_consensus.cc:374] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:25.818734 11226 raft_consensus.cc:729] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcac482cc3db41d99395219dcd0bf95d, State: Initialized, Role: FOLLOWER
I20260619 13:08:25.819363 11208 heartbeater.cc:503] Master 127.0.97.62:40431 was elected leader, sending a full tablet report...
I20260619 13:08:25.819448 11226 consensus_queue.cc:260] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.820039 11227 ts_tablet_manager.cc:1428] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:08:25.820291 11228 raft_consensus.cc:348] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.821137 11228 raft_consensus.cc:374] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:08:25.821249 11228 raft_consensus.cc:729] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb81076d591497d9299f486e71688aa, State: Initialized, Role: FOLLOWER
I20260619 13:08:25.822019 11228 consensus_queue.cc:260] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:25.824179 11228 ts_tablet_manager.cc:1428] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 13:08:25.825207 11074 heartbeater.cc:503] Master 127.0.97.62:40431 was elected leader, sending a full tablet report...
I20260619 13:08:25.826328 11226 ts_tablet_manager.cc:1428] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
W20260619 13:08:25.904506 10941 tablet.cc:2369] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:08:25.929687 11075 tablet.cc:2369] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:08:25.946290 11209 tablet.cc:2369] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:08:26.069154 11233 raft_consensus.cc:482] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:08:26.069463 11233 raft_consensus.cc:504] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:26.071301 11233 leader_election.cc:288] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 701e17423ac8498086eaeee9148c482f (127.0.97.3:33815), 4bb81076d591497d9299f486e71688aa (127.0.97.1:36845)
I20260619 13:08:26.079818 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.071826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56877 (local address 127.0.97.3:33815)
0619 13:08:26.072073 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.072080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.072417 (+ 337us) server_negotiation.cc:407] Connection header received
0619 13:08:26.072707 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.072715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.072810 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.072914 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:08:26.073630 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.074627 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.077289 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.078605 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.078749 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.078859 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.079413 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.079548 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:08:26.079814 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.072679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36207 (local address 127.0.97.1:36845)
0619 13:08:26.072910 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.072916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.073236 (+ 320us) server_negotiation.cc:407] Connection header received
0619 13:08:26.073348 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.073353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.073436 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.073550 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:08:26.074155 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.075014 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.077334 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.078605 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.078748 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.078859 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.079413 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.079548 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 13:08:26.080046 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.071641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33815 (local address 127.0.97.2:56877)
0619 13:08:26.072296 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:26.072313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:26.072488 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:26.073015 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:26.073019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:26.073051 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:08:26.073460 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:26.073475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.074769 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.074776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:26.077051 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:26.077068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.078909 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.078917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:26.079178 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.079276 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 13:08:26.079373 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:08:26.079562 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":98,"thread_start_us":354,"threads_started":1}
I20260619 13:08:26.080454 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.072499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36845 (local address 127.0.97.2:36207)
0619 13:08:26.073122 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 13:08:26.073146 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:08:26.073245 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:08:26.073590 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:08:26.073594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:08:26.073642 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:08:26.074006 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:26.074017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.075142 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.075148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:26.077167 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:08:26.077181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.078912 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.078917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:08:26.079178 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.079272 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 13:08:26.079373 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 13:08:26.079562 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":102,"thread_start_us":284,"threads_started":1}
I20260619 13:08:26.081295 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f70b420e0c4280811e0b51c47e096e" candidate_uuid: "dcac482cc3db41d99395219dcd0bf95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701e17423ac8498086eaeee9148c482f" is_pre_election: true
I20260619 13:08:26.081374 10895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f70b420e0c4280811e0b51c47e096e" candidate_uuid: "dcac482cc3db41d99395219dcd0bf95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb81076d591497d9299f486e71688aa" is_pre_election: true
I20260619 13:08:26.081935 11163 raft_consensus.cc:2454] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac482cc3db41d99395219dcd0bf95d in term 0.
I20260619 13:08:26.081935 10895 raft_consensus.cc:2454] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac482cc3db41d99395219dcd0bf95d in term 0.
I20260619 13:08:26.082645 10962 leader_election.cc:302] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb81076d591497d9299f486e71688aa, dcac482cc3db41d99395219dcd0bf95d; no voters:
I20260619 13:08:26.083016 11233 raft_consensus.cc:2781] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:08:26.083181 11233 raft_consensus.cc:482] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:08:26.083290 11233 raft_consensus.cc:3037] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:26.085491 11233 raft_consensus.cc:504] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:26.086134 11233 leader_election.cc:288] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [CANDIDATE]: Term 1 election: Requested vote from peers 701e17423ac8498086eaeee9148c482f (127.0.97.3:33815), 4bb81076d591497d9299f486e71688aa (127.0.97.1:36845)
I20260619 13:08:26.086654 10895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f70b420e0c4280811e0b51c47e096e" candidate_uuid: "dcac482cc3db41d99395219dcd0bf95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb81076d591497d9299f486e71688aa"
I20260619 13:08:26.086654 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f70b420e0c4280811e0b51c47e096e" candidate_uuid: "dcac482cc3db41d99395219dcd0bf95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701e17423ac8498086eaeee9148c482f"
I20260619 13:08:26.086869 11163 raft_consensus.cc:3037] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:26.086869 10895 raft_consensus.cc:3037] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:08:26.090118 10895 raft_consensus.cc:2454] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac482cc3db41d99395219dcd0bf95d in term 1.
I20260619 13:08:26.090104 11163 raft_consensus.cc:2454] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac482cc3db41d99395219dcd0bf95d in term 1.
I20260619 13:08:26.090572 10962 leader_election.cc:302] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701e17423ac8498086eaeee9148c482f, dcac482cc3db41d99395219dcd0bf95d; no voters:
I20260619 13:08:26.090883 11233 raft_consensus.cc:2781] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:08:26.091341 11233 raft_consensus.cc:686] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 1 LEADER]: Becoming Leader. State: Replica: dcac482cc3db41d99395219dcd0bf95d, State: Running, Role: LEADER
I20260619 13:08:26.091816 11233 consensus_queue.cc:237] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } }
I20260619 13:08:26.096725 10751 catalog_manager.cc:5495] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d reported cstate change: term changed from 0 to 1, leader changed from <none> to dcac482cc3db41d99395219dcd0bf95d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dcac482cc3db41d99395219dcd0bf95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcac482cc3db41d99395219dcd0bf95d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 } health_report { overall_health: UNKNOWN } } }
I20260619 13:08:26.144640 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.138292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.0.97.2:33691)
0619 13:08:26.138512 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.138517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.138539 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:08:26.138676 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.138681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.138756 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.138918 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:08:26.139325 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.140044 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.141420 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.142168 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.142322 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.143057 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:08:26.143930 (+ 873us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:08:26.144101 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.144223 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.144322 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:08:26.153690 10895 raft_consensus.cc:1261] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 1 FOLLOWER]: Refusing update from remote peer dcac482cc3db41d99395219dcd0bf95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:08:26.153934 11163 raft_consensus.cc:1261] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 1 FOLLOWER]: Refusing update from remote peer dcac482cc3db41d99395219dcd0bf95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:08:26.154737 11237 consensus_queue.cc:1035] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb81076d591497d9299f486e71688aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:08:26.155217 11233 consensus_queue.cc:1035] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "701e17423ac8498086eaeee9148c482f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:08:26.170347 11240 mvcc.cc:205] Tried to move back new op lower bound from 7298557977195315200 to 7298557976955994112. Current Snapshot: MvccSnapshot[applied={T|T < 7298557977195315200}]
I20260619 13:08:26.189989 10751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:08:26.190291 10751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:08:26.194494 10751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856dfac99d89449d8fb395d662d21140: Sending DeleteTablet for 3 replicas of tablet 06f70b420e0c4280811e0b51c47e096e
I20260619 13:08:26.195891 11143 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f70b420e0c4280811e0b51c47e096e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:08:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60344
I20260619 13:08:26.196177 11009 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f70b420e0c4280811e0b51c47e096e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:08:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260619 13:08:26.196695 10875 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f70b420e0c4280811e0b51c47e096e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:08:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45082
I20260619 13:08:26.199096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10812
I20260619 13:08:26.202896 11252 tablet_replica.cc:330] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: stopping tablet replica
I20260619 13:08:26.203528 11252 raft_consensus.cc:2229] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:08:26.204226 11252 raft_consensus.cc:2258] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:08:26.205202 11251 tablet_replica.cc:330] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: stopping tablet replica
I20260619 13:08:26.205658 11251 raft_consensus.cc:2229] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:08:26.206442 11251 raft_consensus.cc:2258] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:08:26.207417 11252 ts_tablet_manager.cc:1905] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:08:26.208994 11251 ts_tablet_manager.cc:1905] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:08:26.218254 11252 ts_tablet_manager.cc:1918] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:08:26.218519 11252 log.cc:1196] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/06f70b420e0c4280811e0b51c47e096e
I20260619 13:08:26.219244 11252 ts_tablet_manager.cc:1939] T 06f70b420e0c4280811e0b51c47e096e P 4bb81076d591497d9299f486e71688aa: Deleting consensus metadata
I20260619 13:08:26.221897 10735 catalog_manager.cc:4841] TS 4bb81076d591497d9299f486e71688aa (127.0.97.1:36845): tablet 06f70b420e0c4280811e0b51c47e096e (table test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]) successfully deleted
I20260619 13:08:26.224598 11251 ts_tablet_manager.cc:1918] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:08:26.224874 11251 log.cc:1196] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/06f70b420e0c4280811e0b51c47e096e
I20260619 13:08:26.225562 11251 ts_tablet_manager.cc:1939] T 06f70b420e0c4280811e0b51c47e096e P dcac482cc3db41d99395219dcd0bf95d: Deleting consensus metadata
I20260619 13:08:26.227545 10736 catalog_manager.cc:4841] TS dcac482cc3db41d99395219dcd0bf95d (127.0.97.2:33691): tablet 06f70b420e0c4280811e0b51c47e096e (table test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]) successfully deleted
I20260619 13:08:26.228411 11250 tablet_replica.cc:330] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: stopping tablet replica
I20260619 13:08:26.228899 11250 raft_consensus.cc:2229] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:08:26.229413 11250 raft_consensus.cc:2258] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:08:26.231382 11250 ts_tablet_manager.cc:1905] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:08:26.238592 11250 ts_tablet_manager.cc:1918] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:08:26.238771 11250 log.cc:1196] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/06f70b420e0c4280811e0b51c47e096e
I20260619 13:08:26.239272 11250 ts_tablet_manager.cc:1939] T 06f70b420e0c4280811e0b51c47e096e P 701e17423ac8498086eaeee9148c482f: Deleting consensus metadata
I20260619 13:08:26.240479 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.202172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.97.1:36845)
0619 13:08:26.202360 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.202366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.218185 (+ 15819us) server_negotiation.cc:407] Connection header received
0619 13:08:26.218910 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.218917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.219005 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.219082 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:26.223871 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.225639 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.228305 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.228894 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.229085 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.233955 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:26.233970 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:26.233986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:26.234033 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:26.237177 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.238082 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.238091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.238094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.238241 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.238674 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.238680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.238682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.239150 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:26.239488 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.239852 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.240088 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:08:26.241261 10935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:26.241408 10735 catalog_manager.cc:4841] TS 701e17423ac8498086eaeee9148c482f (127.0.97.3:33815): tablet 06f70b420e0c4280811e0b51c47e096e (table test-table [id=3a6b37cfc006440f8acc45f9b5ddc0f2]) successfully deleted
I20260619 13:08:26.367933 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10812
I20260619 13:08:26.385710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10946
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:08:26.402809 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.386577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.0.97.2:33691)
0619 13:08:26.386761 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.386767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.386835 (+ 68us) server_negotiation.cc:407] Connection header received
0619 13:08:26.386986 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.386991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.387070 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.387194 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:26.388378 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.389781 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.390996 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.391679 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.391821 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.395978 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:26.395997 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:26.396026 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:26.396072 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:26.399944 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.400542 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.400551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.400555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.400719 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.401049 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.401054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.401059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.401510 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:26.401855 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.402187 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.402406 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:08:26.403642 11069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:26.517324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10946
I20260619 13:08:26.536223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 11080
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[10706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:08:26.550652 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:08:26.536985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.97.3:33815)
0619 13:08:26.537155 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:08:26.537160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:08:26.537260 (+ 100us) server_negotiation.cc:407] Connection header received
0619 13:08:26.537410 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:08:26.537414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:08:26.537476 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:08:26.537564 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:08:26.538980 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.539674 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.540905 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:08:26.541281 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:08:26.541370 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:08:26.545659 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:08:26.545676 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:08:26.545693 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:08:26.545744 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:08:26.548286 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.548882 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.548889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.548891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.549027 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:08:26.549333 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:08:26.549336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:08:26.549338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:08:26.549769 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:08:26.549982 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:08:26.550252 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 13:08:26.550406 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:08:26.551329 11203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:26.647706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 11080
I20260619 13:08:26.664526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 10719
I20260619 13:08:26.665719 10781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:08:26.776079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 10719
2026-06-19T13:08:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4555 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[11260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[11260](info): set up 2 sockets
Jun 19 13:08:26 dist-test-slave-q11v krb5kdc[11260](info): commencing operation
krb5kdc: starting...
W20260619 13:08:27.902721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:08:28 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:08:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 13:08:28.246840 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwPkIj_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/postgres -l logfile start
2026-06-19 13:08:30.747 UTC [11290] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:08:30.747 UTC [11290] LOG: listening on IPv4 address "127.0.97.20", port 56499
2026-06-19 13:08:30.751 UTC [11290] LOG: listening on Unix socket "/tmp/.s.PGSQL.56499"
2026-06-19 13:08:30.777 UTC [11293] LOG: database system was shut down at 2026-06-19 13:08:29 UTC
2026-06-19 13:08:30.781 UTC [11290] LOG: database system is ready to accept connections
I20260619 13:08:31.766175 388 mini_postgres.cc:96] Postgres bound to 56499
2026-06-19 13:08:31.783 UTC [11301] FATAL: database "slave" does not exist
127.0.97.20:56499 - accepting connections
I20260619 13:08:31.784613 388 mini_ranger.cc:161] Starting Ranger...
I20260619 13:08:31.810748 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:08:32.259684 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:08:32.259802 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin
2026-06-19 13:08:32,554 [I] DB FLAVOR :POSTGRES
2026-06-19 13:08:32,555 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:08:32,555 [I] Checking connection..
2026-06-19 13:08:32,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:08:32,804 [I] Checking connection passed.
2026-06-19 13:08:32,804 [I] --------- Verifying version history table ---------
2026-06-19 13:08:32,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:08:33,035 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:08:33,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:08:33,296 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:08:33,296 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:08:33,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:08:33.546 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 13:08:33,552 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:08:33,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:08:33,818 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:08:33,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:08:33,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:08:34,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:08:34,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q11v.c.gcp-upstream.internal','N') ;"
2026-06-19 13:08:34,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:08:34,799 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:08:34,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:08:35,059 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:08:35,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:08:35,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:08:35,554 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:08:35,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:08:36.552 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.568 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.584 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.601 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.617 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.634 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.689 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.696 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.711 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.722 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.732 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.746 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.756 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.763 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.769 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:36.775 UTC [11473] WARNING: there is no transaction in progress
2026-06-19 13:08:37,438 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:08:37,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:08:37,679 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:08:37,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:08:37,959 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:08:37,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:08:38,213 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:08:38,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:08:38,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:08:38,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q11v.c.gcp-upstream.internal';"
2026-06-19 13:08:38,987 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:08:38,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:08:39,241 [I] DB_PATCHES have already been applied
I20260619 13:08:39.247155 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwPkIj_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:08:39.247288 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 13:08:39.262707 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:60963
Jun 19, 2026 1:08:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:08:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:08:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:08:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:08:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:08:40.440783 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.656s user 0.000s sys 0.042s
Jun 19, 2026 1:09:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:09:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:09:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:09:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:09:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:09:03.192093 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:09:03 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874543, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:09:03.375747 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:09:03.400579 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:09:04.200532 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:09:04.200734 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms
I20260619 13:09:04.200851 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:56499
2026-06-19 13:09:04,639 [I] DB FLAVOR :POSTGRES
2026-06-19 13:09:04,639 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:09:04,639 [I] Checking connection
2026-06-19 13:09:04,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:09:04,889 [I] connection success
2026-06-19 13:09:04,889 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:09:04,889 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:09:04,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:09:05,151 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:09:05,151 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:09:05,151 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:09:05,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:09:05,427 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:09:05.894932 388 mini_ranger_kms.cc:310] Created kms service
I20260619 13:09:06.075456 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:09:06.124694 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:09:06 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874543, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:06.588655 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:09:06.589059 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:09:06.589133 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 13:09:06.606897 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11654
I20260619 13:09:06.607134 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:54365
Jun 19, 2026 1:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:09:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:09:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 13:09:08 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874548, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:09:08.902257 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.527s user 0.140s sys 0.321s
I20260619 13:09:08.902537 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:09:08.902758 388 mini_ranger_kms.cc:401] 127.0.97.20:54365/kms/v1/keys
Jun 19 13:09:09 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874548, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:09:09 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874548, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:09:11 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:09:12.052954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39455
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39455
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54365/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:09:12.413502 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:09:12.413851 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:09:12.413942 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:09:12.421036 11704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:09:12.421139 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:09:12.421190 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:09:12.421245 11704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:09:12.421293 11704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:09:12.431969 11704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:54365/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39455
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:09:12.433233 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:09:12.434607 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:09:12.446475 11709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:12.446514 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:12.446553 11710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:09:12.446807 11704 server_base.cc:1034] running on GCE node
I20260619 13:09:12.447880 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:09:12.449414 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:09:12.450668 11704 hybrid_clock.cc:648] HybridClock initialized: now 1781874552450610 us; error 49 us; skew 500 ppm
I20260619 13:09:12.451082 11704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:09:12 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:09:12.454854 11704 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:09:12.456403 11704 webserver.cc:460] Webserver started at http://127.0.97.62:38909/ using document root <none> and password file <none>
I20260619 13:09:12.457041 11704 fs_manager.cc:359] Metadata directory not provided
I20260619 13:09:12.457145 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:09:12.457437 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:09:12 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:12.516069 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "4983639a066841979dca69e3b445ca0f"
format_stamp: "Formatted at 2026-06-19 13:09:12 on dist-test-slave-q11v"
server_key: "ba96c8df8863a019ba89da74ae7d02e6"
server_key_iv: "c720dab81f3841b3990ec81b897e4914"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:12.516929 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "4983639a066841979dca69e3b445ca0f"
format_stamp: "Formatted at 2026-06-19 13:09:12 on dist-test-slave-q11v"
server_key: "ba96c8df8863a019ba89da74ae7d02e6"
server_key_iv: "c720dab81f3841b3990ec81b897e4914"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:12.522627 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:09:12.544847 11719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:12.546674 11704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:09:12.546916 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "4983639a066841979dca69e3b445ca0f"
format_stamp: "Formatted at 2026-06-19 13:09:12 on dist-test-slave-q11v"
server_key: "ba96c8df8863a019ba89da74ae7d02e6"
server_key_iv: "c720dab81f3841b3990ec81b897e4914"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:12.547186 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:09:12.569358 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:09:12.574182 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:09:12.574494 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:09:12.593681 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39455
I20260619 13:09:12.593698 11770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39455 every 8 connection(s)
I20260619 13:09:12.595096 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:09:12.599085 11771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:09:12.601534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 11704
I20260619 13:09:12.601783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:09:12.602427 388 external_mini_cluster.cc:1428] Setting key 90bce2f5a2498a3390a3f05e845728cc
I20260619 13:09:12.612979 11771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: Bootstrap starting.
I20260619 13:09:12.616782 11771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: Neither blocks nor log segments found. Creating new log.
Jun 19 13:09:12 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:09:12.618559 11771 log.cc:824] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:09:12.622434 11771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: No bootstrap required, opened a new log
I20260619 13:09:12.627204 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:12.604585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.97.62:39455)
0619 13:09:12.605366 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:12.605385 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:12.605438 (+ 53us) server_negotiation.cc:407] Connection header received
0619 13:09:12.607749 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:12.607780 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:12.608310 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:12.608780 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:12.609977 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:12.612114 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:12.613403 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:12.614483 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:12.614697 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:12.618518 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:12.618533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:12.618561 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:12.618615 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:12.622622 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:12.623300 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:12.623308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:12.623316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:12.623519 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:12.623899 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:12.623905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:12.623908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:12.624677 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:12.624968 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:12.625292 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 13:09:12.625773 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":321,"threads_started":1}
I20260619 13:09:12.632228 11771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } }
I20260619 13:09:12.632988 11771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:09:12.633078 11771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4983639a066841979dca69e3b445ca0f, State: Initialized, Role: FOLLOWER
I20260619 13:09:12.633730 11771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } }
I20260619 13:09:12.633952 11771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:09:12.634102 11771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:09:12.634284 11771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:09:12.637188 11771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } }
I20260619 13:09:12.637738 11771 leader_election.cc:302] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4983639a066841979dca69e3b445ca0f; no voters:
I20260619 13:09:12.638296 11771 leader_election.cc:288] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:09:12.638432 11776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:09:12.639147 11776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [term 1 LEADER]: Becoming Leader. State: Replica: 4983639a066841979dca69e3b445ca0f, State: Running, Role: LEADER
I20260619 13:09:12.639840 11776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } }
I20260619 13:09:12.640671 11771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:09:12.642995 11778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4983639a066841979dca69e3b445ca0f. Latest consensus state: current_term: 1 leader_uuid: "4983639a066841979dca69e3b445ca0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } } }
I20260619 13:09:12.642970 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4983639a066841979dca69e3b445ca0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4983639a066841979dca69e3b445ca0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39455 } } }
I20260619 13:09:12.643416 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [sys.catalog]: This master's current role is: LEADER
I20260619 13:09:12.643416 11778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f [sys.catalog]: This master's current role is: LEADER
I20260619 13:09:12.646780 11783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:09:12.651839 11783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:09:12.659736 11771 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:09:12.670903 11783 catalog_manager.cc:1347] Generated new cluster ID: d22c3a249da446098026d3d0000bccf1
I20260619 13:09:12.671277 11783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:09:12.684340 11783 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:09:12.686403 11783 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:09:12.707589 11783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: Generated new TSK 0
I20260619 13:09:12.708933 11783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:09:13.493479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:09:13 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:09:13.971221 11816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:09:13.971782 11816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:09:13.972441 11816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:09:13.984627 11816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:09:13.984795 11816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:09:13.985044 11816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:09:14.001444 11816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:54365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:09:14.003526 11816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:09:14.005350 11816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:09:14.021857 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:14.021901 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:14.027099 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:09:14.027766 11816 server_base.cc:1034] running on GCE node
I20260619 13:09:14.028527 11816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:09:14.029732 11816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:09:14.030992 11816 hybrid_clock.cc:648] HybridClock initialized: now 1781874554030907 us; error 75 us; skew 500 ppm
I20260619 13:09:14.031351 11816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:09:14 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:09:14.035322 11816 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:09:14.056089 11816 webserver.cc:460] Webserver started at http://127.0.97.1:43825/ using document root <none> and password file <none>
I20260619 13:09:14.057284 11816 fs_manager.cc:359] Metadata directory not provided
I20260619 13:09:14.057541 11816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:09:14.058070 11816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:09:14 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:14.079609 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "0430d932d49c401e97c1e4615c15729f"
format_stamp: "Formatted at 2026-06-19 13:09:14 on dist-test-slave-q11v"
server_key: "a26348f34f8decf05bc7e402b0125dba"
server_key_iv: "ee2bdeb8fcf4dd01e6816f0357991eb9"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:14.082651 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0430d932d49c401e97c1e4615c15729f"
format_stamp: "Formatted at 2026-06-19 13:09:14 on dist-test-slave-q11v"
server_key: "a26348f34f8decf05bc7e402b0125dba"
server_key_iv: "ee2bdeb8fcf4dd01e6816f0357991eb9"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:14.095422 11816 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.004s
I20260619 13:09:14.134374 11836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:14.136801 11816 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.013s sys 0.000s
I20260619 13:09:14.137252 11816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "0430d932d49c401e97c1e4615c15729f"
format_stamp: "Formatted at 2026-06-19 13:09:14 on dist-test-slave-q11v"
server_key: "a26348f34f8decf05bc7e402b0125dba"
server_key_iv: "ee2bdeb8fcf4dd01e6816f0357991eb9"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:14.137699 11816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:09:14.197081 11816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:09:14.216066 11816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:09:14.216544 11816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:09:14.224303 11816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:09:14.233350 11816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:09:14.233558 11816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 13:09:14.233820 11816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:09:14.233970 11816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:09:14 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:09:14 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:14.896497 11816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34705
I20260619 13:09:14.896536 11950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34705 every 8 connection(s)
I20260619 13:09:14.898427 11816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:09:14.904631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 11816
I20260619 13:09:14.904839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:09:14.905650 388 external_mini_cluster.cc:1428] Setting key 884962d965a7c6da71edce289a387790
Jun 19 13:09:14 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:09:14.959892 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:14.902441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39455 (local address 127.0.97.1:48453)
0619 13:09:14.914944 (+ 12503us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:14.915006 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:14.917399 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:14.921632 (+ 4233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:14.921645 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:14.922298 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:09:14.925836 (+ 3538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:14.925869 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:14.927952 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:14.927958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:14.931985 (+ 4027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:14.932005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:14.934264 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:14.934273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:14.935950 (+ 1677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:14.942453 (+ 6503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:09:14.942492 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:09:14.945530 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:09:14.952493 (+ 6963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:14.952504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:14.952550 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:14.953378 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:14.953902 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:14.953906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:14.953910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:14.954180 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:14.955965 (+ 1785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:09:14.955974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:09:14.957480 (+ 1506us) client_negotiation.cc:769] Sending connection context
0619 13:09:14.957844 (+ 364us) client_negotiation.cc:240] Negotiation successful
0619 13:09:14.958428 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12144,"thread_start_us":11901,"threads_started":1}
I20260619 13:09:14.961088 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:14.901533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48453 (local address 127.0.97.62:39455)
0619 13:09:14.920950 (+ 19417us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:14.920957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:14.920980 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:09:14.921049 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:14.921055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:14.921140 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:14.921293 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:14.926065 (+ 4772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:14.927493 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:14.932235 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:14.933973 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:14.945706 (+ 11733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:14.947585 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:14.947593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:14.947598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:14.947643 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:14.952290 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:14.953542 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:14.953549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:14.953552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:14.953756 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:14.955310 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:14.955316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:14.955320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:14.955722 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:14.955863 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:14.960479 (+ 4616us) server_negotiation.cc:299] Negotiation successful
0619 13:09:14.960731 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19265,"thread_start_us":19876,"threads_started":1}
I20260619 13:09:14.962551 11951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39455
I20260619 13:09:14.962891 11951 heartbeater.cc:464] Registering TS with master...
I20260619 13:09:14.963734 11951 heartbeater.cc:511] Master 127.0.97.62:39455 requested a full tablet report, sending...
I20260619 13:09:14.966287 11736 ts_manager.cc:194] Registered new tserver with Master: 0430d932d49c401e97c1e4615c15729f (127.0.97.1:34705)
I20260619 13:09:14.968283 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:09:14.997053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:09:15.370540 11957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:09:15.370864 11957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:09:15.371069 11957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:09:15.379258 11957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:09:15.379418 11957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:09:15.379611 11957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:09:15.390275 11957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:54365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:09:15.391650 11957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:09:15.393023 11957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:09:15.408684 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:15.408886 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:15.409117 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:09:15.409752 11957 server_base.cc:1034] running on GCE node
I20260619 13:09:15.410496 11957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:09:15.411654 11957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:09:15.412896 11957 hybrid_clock.cc:648] HybridClock initialized: now 1781874555412826 us; error 59 us; skew 500 ppm
I20260619 13:09:15.413264 11957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:09:15 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:09:15.417801 11957 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:09:15.419692 11957 webserver.cc:460] Webserver started at http://127.0.97.2:35163/ using document root <none> and password file <none>
I20260619 13:09:15.420573 11957 fs_manager.cc:359] Metadata directory not provided
I20260619 13:09:15.420725 11957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:09:15.421085 11957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:09:15 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:15.441174 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "7497680ecd9a42af9db94ff0e201f223"
format_stamp: "Formatted at 2026-06-19 13:09:15 on dist-test-slave-q11v"
server_key: "fc7c918331f7c71e6419d6d17869e48f"
server_key_iv: "4e5142c2ba891c33205268c8bf1dbf8e"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:15.442555 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7497680ecd9a42af9db94ff0e201f223"
format_stamp: "Formatted at 2026-06-19 13:09:15 on dist-test-slave-q11v"
server_key: "fc7c918331f7c71e6419d6d17869e48f"
server_key_iv: "4e5142c2ba891c33205268c8bf1dbf8e"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:15.450802 11957 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.002s
I20260619 13:09:15.470372 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:15.472517 11957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:09:15.472755 11957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "7497680ecd9a42af9db94ff0e201f223"
format_stamp: "Formatted at 2026-06-19 13:09:15 on dist-test-slave-q11v"
server_key: "fc7c918331f7c71e6419d6d17869e48f"
server_key_iv: "4e5142c2ba891c33205268c8bf1dbf8e"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:15.473001 11957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:09:15.497102 11957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:09:15.501482 11957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:09:15.501829 11957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:09:15.502830 11957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:09:15.504197 11957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:09:15.504300 11957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:15.504448 11957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:09:15.504519 11957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:15.542723 11957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36951
I20260619 13:09:15.542740 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36951 every 8 connection(s)
I20260619 13:09:15.544128 11957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:09:15.547719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 11957
I20260619 13:09:15.547936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:09:15.548628 388 external_mini_cluster.cc:1428] Setting key d656bba91bdded344e33fcfb5243cea5
Jun 19 13:09:15 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:09:15.570043 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:15.546983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33239 (local address 127.0.97.62:39455)
0619 13:09:15.547471 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:15.547477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:15.552823 (+ 5346us) server_negotiation.cc:407] Connection header received
0619 13:09:15.552894 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:15.552902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:15.552995 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:15.553102 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:15.555514 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:15.556608 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:15.559214 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:15.559795 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:15.559919 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:15.563934 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:15.563941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:15.563945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:15.563978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:15.566708 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:15.567471 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:15.567475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:15.567477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:15.567599 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:15.568072 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:15.568079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:15.568087 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:15.568372 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:15.568530 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:15.569304 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 13:09:15.569535 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260619 13:09:15.571048 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:15.547632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39455 (local address 127.0.97.2:33239)
0619 13:09:15.549915 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:15.549971 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:15.552443 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:15.553557 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:15.553573 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:15.554299 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:09:15.555193 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:15.555277 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:15.556804 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:15.556812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:15.559059 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:15.559077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:15.559998 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:15.560003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:15.560218 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:15.560965 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:09:15.560990 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:09:15.563513 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:09:15.566857 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:15.566866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:15.566885 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:15.567330 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:15.567717 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:15.567722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:15.567726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:15.567947 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:15.568504 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:09:15.568512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:09:15.568905 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 13:09:15.569180 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 13:09:15.569756 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":1746,"threads_started":1}
I20260619 13:09:15.572569 12092 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39455
I20260619 13:09:15.572896 12092 heartbeater.cc:464] Registering TS with master...
I20260619 13:09:15.573673 12092 heartbeater.cc:511] Master 127.0.97.62:39455 requested a full tablet report, sending...
I20260619 13:09:15.575106 11736 ts_manager.cc:194] Registered new tserver with Master: 7497680ecd9a42af9db94ff0e201f223 (127.0.97.2:36951)
I20260619 13:09:15.575934 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33239
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:09:15.612622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:09:15.972107 11951 heartbeater.cc:503] Master 127.0.97.62:39455 was elected leader, sending a full tablet report...
W20260619 13:09:15.977726 12098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:09:15.978004 12098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:09:15.978152 12098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:09:15.985486 12098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:09:15.985621 12098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:09:15.985807 12098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:09:15.995921 12098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:54365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:09:15.997416 12098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:09:15.998857 12098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:09:16.011854 12103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:16.011878 12104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:09:16.011802 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:09:16.012884 12098 server_base.cc:1034] running on GCE node
I20260619 13:09:16.013509 12098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:09:16.014568 12098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:09:16.015779 12098 hybrid_clock.cc:648] HybridClock initialized: now 1781874556015734 us; error 39 us; skew 500 ppm
I20260619 13:09:16.016112 12098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:09:16.020093 12098 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:09:16.021708 12098 webserver.cc:460] Webserver started at http://127.0.97.3:35597/ using document root <none> and password file <none>
I20260619 13:09:16.022425 12098 fs_manager.cc:359] Metadata directory not provided
I20260619 13:09:16.022532 12098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:09:16.022898 12098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:16.037787 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "f29bcb625ab94d2d88c294136bf4a4b0"
format_stamp: "Formatted at 2026-06-19 13:09:16 on dist-test-slave-q11v"
server_key: "ff434db8be58a73b779ebf03c4821b81"
server_key_iv: "9c6226dc1e8a3bee452ddfd31014e078"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:16.038745 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f29bcb625ab94d2d88c294136bf4a4b0"
format_stamp: "Formatted at 2026-06-19 13:09:16 on dist-test-slave-q11v"
server_key: "ff434db8be58a73b779ebf03c4821b81"
server_key_iv: "9c6226dc1e8a3bee452ddfd31014e078"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:16.044404 12098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:09:16.061210 12113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:16.062891 12098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:09:16.063097 12098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "f29bcb625ab94d2d88c294136bf4a4b0"
format_stamp: "Formatted at 2026-06-19 13:09:16 on dist-test-slave-q11v"
server_key: "ff434db8be58a73b779ebf03c4821b81"
server_key_iv: "9c6226dc1e8a3bee452ddfd31014e078"
server_key_version: "kuduclusterkey@0"
I20260619 13:09:16.063336 12098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:09:16.095922 12098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:09:16.100461 12098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:09:16.100776 12098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:09:16.101791 12098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:09:16.103173 12098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:09:16.103704 12098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:16.103865 12098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:09:16.103919 12098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:09:16.138739 12098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33609
I20260619 13:09:16.138775 12225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33609 every 8 connection(s)
I20260619 13:09:16.140033 12098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:09:16.140522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 12098
I20260619 13:09:16.140873 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:09:16.141446 388 external_mini_cluster.cc:1428] Setting key d569679294728d115db49529eea831ab
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:09:16.163081 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.142949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48461 (local address 127.0.97.62:39455)
0619 13:09:16.143648 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.143656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.145107 (+ 1451us) server_negotiation.cc:407] Connection header received
0619 13:09:16.148354 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.148367 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.148518 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.148639 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:16.150745 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.151670 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.153791 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.154191 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.154283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.157815 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:16.157821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:16.157824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:16.157855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:16.160516 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.161153 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.161157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.161159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.161275 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.161727 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.161730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.161732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.161980 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:16.162086 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.162686 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.162834 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":733,"threads_started":1}
I20260619 13:09:16.164124 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.143633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39455 (local address 127.0.97.3:48461)
0619 13:09:16.144863 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.144950 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.147409 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.148903 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.148912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.149550 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:09:16.150512 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.150539 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.151829 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.151833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.153588 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.153653 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.154369 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.154373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.154567 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.155177 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:09:16.155197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:09:16.157570 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:09:16.160627 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:16.160633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:16.160645 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:16.161049 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:16.161386 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:09:16.161389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:09:16.161391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:09:16.161584 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:09:16.162089 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:09:16.162095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:09:16.162379 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.162595 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.162955 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":483,"threads_started":1}
I20260619 13:09:16.165470 12226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39455
I20260619 13:09:16.165793 12226 heartbeater.cc:464] Registering TS with master...
I20260619 13:09:16.166481 12226 heartbeater.cc:511] Master 127.0.97.62:39455 requested a full tablet report, sending...
I20260619 13:09:16.167992 11736 ts_manager.cc:194] Registered new tserver with Master: f29bcb625ab94d2d88c294136bf4a4b0 (127.0.97.3:33609)
I20260619 13:09:16.169241 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48461
I20260619 13:09:16.170154 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:09:16.189385 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.174733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.97.62:39455)
0619 13:09:16.174975 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.174984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.175310 (+ 326us) server_negotiation.cc:407] Connection header received
0619 13:09:16.175537 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.175544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.175620 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.175726 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:16.177575 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.178332 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.180028 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.180813 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.181081 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.183811 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:16.183817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:16.183820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:16.183860 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:16.186614 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.187380 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.187387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.187391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.187528 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.187936 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.187942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.187946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.188230 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:16.188488 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.188829 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.189054 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:09:16.193823 11735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:09:16.201040 11735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:09:16.226178 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.214682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.0.1:46876)
0619 13:09:16.216628 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.216700 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.216922 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.217902 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.217912 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.217952 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:09:16.218464 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.218479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.220489 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.220497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.222704 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.222715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.225030 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.225035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.225298 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.225379 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.225542 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.225776 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"mutex_wait_us":40,"thread_start_us":1081,"threads_started":1}
I20260619 13:09:16.226189 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.214801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.0.97.1:34705)
0619 13:09:16.215558 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.215571 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.217142 (+ 1571us) server_negotiation.cc:407] Connection header received
0619 13:09:16.217271 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.217282 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.217626 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.218317 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:09:16.218644 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.220326 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.222866 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.224840 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.224991 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.225240 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.225638 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.225821 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":318,"threads_started":1}
I20260619 13:09:16.226297 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.215337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.0.97.2:36951)
0619 13:09:16.215924 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.215934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.217036 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 13:09:16.217338 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.217347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.217627 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.217802 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:09:16.219311 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.220776 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.223295 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.224723 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.225253 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.225434 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.225786 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.225970 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":652,"threads_started":1}
I20260619 13:09:16.227432 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.216584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33609 (local address 127.0.0.1:35368)
0619 13:09:16.218933 (+ 2349us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.218963 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.219104 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.219698 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.219703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.219727 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:09:16.220045 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.220055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.222008 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.222014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.223942 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.223956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.226574 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.226579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.226747 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.226777 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.226910 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.227037 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2223,"mutex_wait_us":59,"thread_start_us":2161,"threads_started":1}
I20260619 13:09:16.227458 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.217054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.97.3:33609)
0619 13:09:16.218480 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.218488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.219086 (+ 598us) server_negotiation.cc:407] Connection header received
0619 13:09:16.219267 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.219276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.219504 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.219685 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:09:16.220271 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.221866 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.224085 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.225981 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.226105 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.226329 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.226952 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.227105 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":1003,"threads_started":1}
I20260619 13:09:16.228019 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.215730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36951 (local address 127.0.0.1:38998)
0619 13:09:16.216904 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.216921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.217143 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.218174 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.218193 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.218222 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:09:16.219104 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.219114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.220939 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.220944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.223122 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.223137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.225398 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.225403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.225579 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.225630 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.227601 (+ 1971us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.227712 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":53,"thread_start_us":1054,"threads_started":1}
I20260619 13:09:16.230878 11884 tablet_service.cc:1467] Processing CreateTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (DEFAULT_TABLE table=test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:09:16.231474 12161 tablet_service.cc:1467] Processing CreateTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (DEFAULT_TABLE table=test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:09:16.231479 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (DEFAULT_TABLE table=test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:09:16.232236 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:09:16.232857 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:09:16.232857 12161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:09:16.244836 12245 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Bootstrap starting.
I20260619 13:09:16.248728 12245 tablet_bootstrap.cc:654] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Neither blocks nor log segments found. Creating new log.
I20260619 13:09:16.249274 12246 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Bootstrap starting.
I20260619 13:09:16.250252 12247 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Bootstrap starting.
I20260619 13:09:16.250375 12245 log.cc:824] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:09:16.253829 12247 tablet_bootstrap.cc:654] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Neither blocks nor log segments found. Creating new log.
I20260619 13:09:16.254405 12246 tablet_bootstrap.cc:654] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Neither blocks nor log segments found. Creating new log.
I20260619 13:09:16.254873 12247 log.cc:824] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Log is configured to *not* fsync() on all Append() calls
I20260619 13:09:16.255407 12246 log.cc:824] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:09:16.257910 12245 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: No bootstrap required, opened a new log
I20260619 13:09:16.258203 12245 ts_tablet_manager.cc:1397] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260619 13:09:16.258224 12247 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: No bootstrap required, opened a new log
I20260619 13:09:16.258483 12247 ts_tablet_manager.cc:1397] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260619 13:09:16.260305 12246 tablet_bootstrap.cc:492] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: No bootstrap required, opened a new log
I20260619 13:09:16.260565 12246 ts_tablet_manager.cc:1397] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260619 13:09:16.265715 12247 raft_consensus.cc:348] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.266117 12247 raft_consensus.cc:374] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:09:16.266250 12247 raft_consensus.cc:729] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7497680ecd9a42af9db94ff0e201f223, State: Initialized, Role: FOLLOWER
I20260619 13:09:16.266878 12247 consensus_queue.cc:260] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.269675 12092 heartbeater.cc:503] Master 127.0.97.62:39455 was elected leader, sending a full tablet report...
I20260619 13:09:16.269932 12245 raft_consensus.cc:348] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.270385 12245 raft_consensus.cc:374] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:09:16.270545 12245 raft_consensus.cc:729] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0430d932d49c401e97c1e4615c15729f, State: Initialized, Role: FOLLOWER
I20260619 13:09:16.271181 12245 consensus_queue.cc:260] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.271533 12246 raft_consensus.cc:348] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.272054 12246 raft_consensus.cc:374] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:09:16.272110 12247 ts_tablet_manager.cc:1428] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260619 13:09:16.272208 12246 raft_consensus.cc:729] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29bcb625ab94d2d88c294136bf4a4b0, State: Initialized, Role: FOLLOWER
I20260619 13:09:16.272948 12246 consensus_queue.cc:260] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.273968 12245 ts_tablet_manager.cc:1428] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 13:09:16.274873 12226 heartbeater.cc:503] Master 127.0.97.62:39455 was elected leader, sending a full tablet report...
I20260619 13:09:16.275519 12246 ts_tablet_manager.cc:1428] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Time spent starting tablet: real 0.015s user 0.015s sys 0.002s
W20260619 13:09:16.296171 12093 tablet.cc:2369] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:09:16.393007 12227 tablet.cc:2369] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:09:16.407250 11952 tablet.cc:2369] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:09:16.425971 12253 raft_consensus.cc:482] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:09:16.426429 12253 raft_consensus.cc:504] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.428566 12253 leader_election.cc:288] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0430d932d49c401e97c1e4615c15729f (127.0.97.1:34705), 7497680ecd9a42af9db94ff0e201f223 (127.0.97.2:36951)
I20260619 13:09:16.434883 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.429240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37521 (local address 127.0.97.1:34705)
0619 13:09:16.429441 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.429445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.429462 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:09:16.429818 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.429825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.429914 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.430035 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:09:16.430491 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.431254 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.433031 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.433895 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.434067 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.434184 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.434464 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.434607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:09:16.434906 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.428960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.97.3:37521)
0619 13:09:16.429302 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.429325 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.429487 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.430101 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.430107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.430144 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:09:16.430318 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.430327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.431405 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.431413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.432850 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.432863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.434078 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.434094 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.434282 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.434306 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.434437 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.434592 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":28}
I20260619 13:09:16.435192 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.429149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59589 (local address 127.0.97.2:36951)
0619 13:09:16.429357 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.429363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.430142 (+ 779us) server_negotiation.cc:407] Connection header received
0619 13:09:16.430406 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.430411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.430485 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.430584 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:09:16.431078 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.431764 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.433424 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.434285 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.434454 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.434545 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.434809 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.434932 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:09:16.435302 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.429125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36951 (local address 127.0.97.3:59589)
0619 13:09:16.430048 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 13:09:16.430067 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:09:16.430253 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:09:16.430598 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:09:16.430602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:09:16.430633 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:09:16.430926 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.430934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.431921 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.431927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.433278 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:09:16.433289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.434493 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.434500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:09:16.434662 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.434684 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:09:16.434786 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:09:16.434914 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":2,"thread_start_us":544,"threads_started":1}
I20260619 13:09:16.436331 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d63bf2af27f4e7d8feb8b13170c5eb5" candidate_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7497680ecd9a42af9db94ff0e201f223" is_pre_election: true
I20260619 13:09:16.436462 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d63bf2af27f4e7d8feb8b13170c5eb5" candidate_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0430d932d49c401e97c1e4615c15729f" is_pre_election: true
I20260619 13:09:16.436868 12047 raft_consensus.cc:2454] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29bcb625ab94d2d88c294136bf4a4b0 in term 0.
I20260619 13:09:16.437031 11904 raft_consensus.cc:2454] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29bcb625ab94d2d88c294136bf4a4b0 in term 0.
I20260619 13:09:16.437625 12115 leader_election.cc:302] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7497680ecd9a42af9db94ff0e201f223, f29bcb625ab94d2d88c294136bf4a4b0; no voters:
I20260619 13:09:16.438050 12253 raft_consensus.cc:2781] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:09:16.438218 12253 raft_consensus.cc:482] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:09:16.438359 12253 raft_consensus.cc:3037] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:09:16.441155 12253 raft_consensus.cc:504] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.441959 12253 leader_election.cc:288] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [CANDIDATE]: Term 1 election: Requested vote from peers 0430d932d49c401e97c1e4615c15729f (127.0.97.1:34705), 7497680ecd9a42af9db94ff0e201f223 (127.0.97.2:36951)
I20260619 13:09:16.442463 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d63bf2af27f4e7d8feb8b13170c5eb5" candidate_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0430d932d49c401e97c1e4615c15729f"
I20260619 13:09:16.442516 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d63bf2af27f4e7d8feb8b13170c5eb5" candidate_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7497680ecd9a42af9db94ff0e201f223"
I20260619 13:09:16.442694 11904 raft_consensus.cc:3037] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:09:16.442710 12047 raft_consensus.cc:3037] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:09:16.444862 12047 raft_consensus.cc:2454] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29bcb625ab94d2d88c294136bf4a4b0 in term 1.
I20260619 13:09:16.444970 11904 raft_consensus.cc:2454] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29bcb625ab94d2d88c294136bf4a4b0 in term 1.
I20260619 13:09:16.445314 12115 leader_election.cc:302] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7497680ecd9a42af9db94ff0e201f223, f29bcb625ab94d2d88c294136bf4a4b0; no voters:
I20260619 13:09:16.445676 12253 raft_consensus.cc:2781] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:09:16.446141 12253 raft_consensus.cc:686] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 1 LEADER]: Becoming Leader. State: Replica: f29bcb625ab94d2d88c294136bf4a4b0, State: Running, Role: LEADER
I20260619 13:09:16.446697 12253 consensus_queue.cc:237] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } }
I20260619 13:09:16.451342 11735 catalog_manager.cc:5495] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f29bcb625ab94d2d88c294136bf4a4b0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29bcb625ab94d2d88c294136bf4a4b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 } health_report { overall_health: UNKNOWN } } }
I20260619 13:09:16.467453 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.461850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.0.97.3:33609)
0619 13:09:16.462027 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.462032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.462049 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:09:16.462171 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.462175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.462254 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.462387 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:09:16.462743 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.463446 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.465339 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.465775 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.465871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.466353 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:09:16.466847 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:09:16.466955 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.467104 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.467166 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:09:16.475592 12047 raft_consensus.cc:1261] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 1 FOLLOWER]: Refusing update from remote peer f29bcb625ab94d2d88c294136bf4a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:09:16.475663 11904 raft_consensus.cc:1261] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 1 FOLLOWER]: Refusing update from remote peer f29bcb625ab94d2d88c294136bf4a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:09:16.476601 12255 consensus_queue.cc:1035] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0430d932d49c401e97c1e4615c15729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:09:16.476972 12253 consensus_queue.cc:1035] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7497680ecd9a42af9db94ff0e201f223" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:09:16.504479 11736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:09:16.504699 11736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:09:16.508440 11736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4983639a066841979dca69e3b445ca0f: Sending DeleteTablet for 3 replicas of tablet 8d63bf2af27f4e7d8feb8b13170c5eb5
I20260619 13:09:16.509712 11884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46876
I20260619 13:09:16.509958 12161 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35368
I20260619 13:09:16.510520 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:09:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38998
I20260619 13:09:16.511443 12270 tablet_replica.cc:330] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: stopping tablet replica
I20260619 13:09:16.513101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 11816
I20260619 13:09:16.520767 12268 tablet_replica.cc:330] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: stopping tablet replica
I20260619 13:09:16.521104 12269 tablet_replica.cc:330] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: stopping tablet replica
I20260619 13:09:16.521271 12268 raft_consensus.cc:2229] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:09:16.521682 12269 raft_consensus.cc:2229] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:09:16.521857 12268 raft_consensus.cc:2258] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:09:16.522110 12270 raft_consensus.cc:2229] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:09:16.522670 12269 raft_consensus.cc:2258] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:09:16.524220 12268 ts_tablet_manager.cc:1905] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:09:16.524698 12269 ts_tablet_manager.cc:1905] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:09:16.531448 12269 ts_tablet_manager.cc:1918] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:09:16.531448 12268 ts_tablet_manager.cc:1918] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:09:16.531610 12268 log.cc:1196] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/8d63bf2af27f4e7d8feb8b13170c5eb5
I20260619 13:09:16.531606 12269 log.cc:1196] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/8d63bf2af27f4e7d8feb8b13170c5eb5
I20260619 13:09:16.532122 12269 ts_tablet_manager.cc:1939] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P f29bcb625ab94d2d88c294136bf4a4b0: Deleting consensus metadata
I20260619 13:09:16.532122 12268 ts_tablet_manager.cc:1939] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 0430d932d49c401e97c1e4615c15729f: Deleting consensus metadata
I20260619 13:09:16.533999 11723 catalog_manager.cc:4841] TS f29bcb625ab94d2d88c294136bf4a4b0 (127.0.97.3:33609): tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (table test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]) successfully deleted
I20260619 13:09:16.523627 12270 raft_consensus.cc:2258] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:09:16.534373 11723 catalog_manager.cc:4841] TS 0430d932d49c401e97c1e4615c15729f (127.0.97.1:34705): tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (table test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]) successfully deleted
I20260619 13:09:16.537310 12270 ts_tablet_manager.cc:1905] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:09:16.544530 12270 ts_tablet_manager.cc:1918] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:09:16.544724 12270 log.cc:1196] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/8d63bf2af27f4e7d8feb8b13170c5eb5
I20260619 13:09:16.545234 12270 ts_tablet_manager.cc:1939] T 8d63bf2af27f4e7d8feb8b13170c5eb5 P 7497680ecd9a42af9db94ff0e201f223: Deleting consensus metadata
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:09:16.547138 11721 catalog_manager.cc:4841] TS 7497680ecd9a42af9db94ff0e201f223 (127.0.97.2:36951): tablet 8d63bf2af27f4e7d8feb8b13170c5eb5 (table test-table [id=8aac5417cb9c4b65bb5a7f3c099ab93d]) successfully deleted
I20260619 13:09:16.556306 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.516120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.0.97.1:34705)
0619 13:09:16.516306 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.516313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.536167 (+ 19854us) server_negotiation.cc:407] Connection header received
0619 13:09:16.536359 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.536366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.536459 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.536584 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:16.538312 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.539515 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.541501 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.542175 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.542281 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.548873 (+ 6592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:16.548892 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:16.548910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:16.548955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:16.552943 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.553926 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.553936 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.553940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.554093 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.554548 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.554552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.554555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.555012 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:16.555297 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.555662 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.555920 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:09:16.557090 11946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:09:16.681310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 11816
I20260619 13:09:16.699559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 11957
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:09:16.716627 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.700447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.0.97.2:36951)
0619 13:09:16.700639 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.700645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.700666 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:09:16.700801 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.700812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.700900 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.701027 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:16.702352 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.703105 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.704353 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.704734 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.704884 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.709415 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:16.709434 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:16.709455 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:16.709500 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:16.713424 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.714266 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.714277 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.714281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.714464 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.714853 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.714858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.714862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.715385 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:16.715696 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.715962 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.716193 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:09:16.717538 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:09:16.807968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 11957
I20260619 13:09:16.824724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 12098
Jun 19 13:09:16 dist-test-slave-q11v krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:09:16.842672 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:09:16.825544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.97.3:33609)
0619 13:09:16.825776 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 13:09:16.825782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:09:16.825807 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:09:16.825945 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:09:16.825951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:09:16.826032 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:09:16.826147 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:09:16.827411 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.828589 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.830085 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:09:16.830810 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:09:16.830953 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:09:16.835458 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:09:16.835473 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:09:16.835493 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:09:16.835571 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:09:16.839645 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.840429 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.840436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.840438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.840597 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:09:16.841057 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:09:16.841060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:09:16.841062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:09:16.841510 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:09:16.841826 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 13:09:16.842150 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 13:09:16.842334 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 13:09:16.843400 12221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:09:16.966329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 12098
I20260619 13:09:16.986025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 11704
I20260619 13:09:16.987563 11766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:09:17.100296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 11704
I20260619 13:09:17.124658 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:09:17.460307 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:09:17.460567 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:09:17.806252 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:09:17.806 UTC [11290] LOG: received smart shutdown request
2026-06-19 13:09:17.810 UTC [11290] LOG: background worker "logical replication launcher" (PID 11299) exited with exit code 1
2026-06-19 13:09:17.810 UTC [11294] LOG: shutting down
2026-06-19 13:09:17.967 UTC [11290] LOG: database system is shut down
2026-06-19T13:09:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51311 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:09:18 dist-test-slave-q11v krb5kdc[12284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:09:18 dist-test-slave-q11v krb5kdc[12284](info): set up 2 sockets
Jun 19 13:09:18 dist-test-slave-q11v krb5kdc[12284](info): commencing operation
krb5kdc: starting...
W20260619 13:09:19.241945 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:09:19 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:09:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 13:09:19.572167 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwPkIj_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/postgres -l logfile start
2026-06-19 13:09:22.014 UTC [12314] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:09:22.014 UTC [12314] LOG: listening on IPv4 address "127.0.97.20", port 59199
2026-06-19 13:09:22.018 UTC [12314] LOG: listening on Unix socket "/tmp/.s.PGSQL.59199"
2026-06-19 13:09:22.040 UTC [12317] LOG: database system was shut down at 2026-06-19 13:09:20 UTC
2026-06-19 13:09:22.044 UTC [12314] LOG: database system is ready to accept connections
I20260619 13:09:23.017737 388 mini_postgres.cc:96] Postgres bound to 59199
2026-06-19 13:09:23.034 UTC [12325] FATAL: database "slave" does not exist
127.0.97.20:59199 - accepting connections
I20260619 13:09:23.035538 388 mini_ranger.cc:161] Starting Ranger...
I20260619 13:09:23.062245 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:09:23.510006 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:09:23.510131 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin
2026-06-19 13:09:23,805 [I] DB FLAVOR :POSTGRES
2026-06-19 13:09:23,806 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:09:23,806 [I] Checking connection..
2026-06-19 13:09:23,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:09:24,054 [I] Checking connection passed.
2026-06-19 13:09:24,055 [I] --------- Verifying version history table ---------
2026-06-19 13:09:24,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:09:24,297 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:09:24,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:09:24,542 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:09:24,542 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:09:24,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:09:24.789 UTC [12389] WARNING: there is no transaction in progress
2026-06-19 13:09:24,795 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:09:24,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:09:25,041 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:09:25,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:09:25,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:09:25,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:09:25,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q11v.c.gcp-upstream.internal','N') ;"
2026-06-19 13:09:25,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:09:26,022 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:09:26,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:09:26,282 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:09:26,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:09:26,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:09:26,798 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:09:26,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:09:27.764 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.780 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.797 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.813 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.831 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.850 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.915 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.922 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.936 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.946 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.957 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.972 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.982 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.989 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:27.996 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:28.003 UTC [12497] WARNING: there is no transaction in progress
2026-06-19 13:09:28,658 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:09:28,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:09:28,897 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:09:28,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:09:29,160 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:09:29,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:09:29,403 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:09:29,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:09:29,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:09:29,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q11v.c.gcp-upstream.internal';"
2026-06-19 13:09:30,129 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:09:30,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:09:30,399 [I] DB_PATCHES have already been applied
I20260619 13:09:30.405472 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwPkIj_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:09:30.405557 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 13:09:30.420966 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:58967
Jun 19, 2026 1:09:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:09:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:09:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:09:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:09:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:09:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:09:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:09:31.559247 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.524s user 0.002s sys 0.036s
Jun 19, 2026 1:09:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:09:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:09:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:09:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:09:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:09:54.138769 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:09:54 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:09:54.302704 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:09:54.323939 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:09:55.162024 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:09:55.162148 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms
I20260619 13:09:55.162218 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:59199
2026-06-19 13:09:55,514 [I] DB FLAVOR :POSTGRES
2026-06-19 13:09:55,514 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:09:55,514 [I] Checking connection
2026-06-19 13:09:55,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:09:55,756 [I] connection success
2026-06-19 13:09:55,756 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:09:55,756 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:09:55,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:09:56,022 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:09:56,023 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:09:56,023 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:09:56,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:09:56,284 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:09:56.758622 388 mini_ranger_kms.cc:310] Created kms service
I20260619 13:09:56.965966 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:09:57.016650 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:09:57 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:09:57.427439 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:09:57.427800 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwPkIj_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwPkIj_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:09:57.427856 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 13:09:57.443993 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12678
I20260619 13:09:57.444139 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:57295
Jun 19, 2026 1:09:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:09:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:09:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:09:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:09:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:09:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 13:09:59.731690 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.429s user 0.094s sys 0.270s
I20260619 13:09:59.731940 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:09:59.732112 388 mini_ranger_kms.cc:401] 127.0.97.20:57295/kms/v1/keys
Jun 19 13:09:59 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 13:10:00 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:10:00 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:10:03 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:10:03.075205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36467
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36467
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:03.409169 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:03.409482 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:03.409569 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:03.417009 12728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:03.417172 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:03.417232 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:03.417291 12728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:03.417338 12728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:03.428328 12728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:57295/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36467
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:03.429657 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:03.430963 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:03.443120 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:03.443092 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:03.443325 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:03.443729 12728 server_base.cc:1034] running on GCE node
I20260619 13:10:03.444581 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:03.446106 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:03.447340 12728 hybrid_clock.cc:648] HybridClock initialized: now 1781874603447270 us; error 55 us; skew 500 ppm
I20260619 13:10:03.447676 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:03 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:03.455061 12728 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:03.456709 12728 webserver.cc:460] Webserver started at http://127.0.97.62:33961/ using document root <none> and password file <none>
I20260619 13:10:03.457412 12728 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:03.457533 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:03.457894 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:10:03 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:03.516363 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "0b77a355a03d4d0b89591b6707924c44"
format_stamp: "Formatted at 2026-06-19 13:10:03 on dist-test-slave-q11v"
server_key: "8a140398d6be90031ff40b473afc03d4"
server_key_iv: "2498df2e8c5d4cb978ddd414bfd1094b"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:03.517217 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b77a355a03d4d0b89591b6707924c44"
format_stamp: "Formatted at 2026-06-19 13:10:03 on dist-test-slave-q11v"
server_key: "8a140398d6be90031ff40b473afc03d4"
server_key_iv: "2498df2e8c5d4cb978ddd414bfd1094b"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:03.522900 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:10:03.543962 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:03.546424 12728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 13:10:03.546685 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "0b77a355a03d4d0b89591b6707924c44"
format_stamp: "Formatted at 2026-06-19 13:10:03 on dist-test-slave-q11v"
server_key: "8a140398d6be90031ff40b473afc03d4"
server_key_iv: "2498df2e8c5d4cb978ddd414bfd1094b"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:03.546964 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:03.560315 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:03.564538 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:03.564873 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:03.583356 12728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36467
I20260619 13:10:03.583396 12794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36467 every 8 connection(s)
I20260619 13:10:03.584714 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:03.588660 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:03.590351 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 12728
I20260619 13:10:03.590587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:10:03.591246 388 external_mini_cluster.cc:1428] Setting key a03e29b2fc94ba2935de216d10d629fe
I20260619 13:10:03.600764 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Bootstrap starting.
I20260619 13:10:03.604696 12795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Neither blocks nor log segments found. Creating new log.
Jun 19 13:10:03 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:03.605866 12795 log.cc:824] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:03.608794 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: No bootstrap required, opened a new log
I20260619 13:10:03.613700 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:03.593147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.97.62:36467)
0619 13:10:03.594052 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:03.594064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:03.594106 (+ 42us) server_negotiation.cc:407] Connection header received
0619 13:10:03.595718 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:03.595740 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:03.596126 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:03.596443 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:03.598055 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:03.599363 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:03.601134 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:03.601832 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:03.601979 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:03.606543 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:03.606550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:03.606569 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:03.606607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:03.609721 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:03.610373 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:03.610388 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:03.610395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:03.610565 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:03.610927 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:03.610934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:03.610939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:03.611473 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:03.611790 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:03.612225 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 13:10:03.612629 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":440,"threads_started":1}
I20260619 13:10:03.616717 12795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:03.617090 12795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:03.617190 12795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b77a355a03d4d0b89591b6707924c44, State: Initialized, Role: FOLLOWER
I20260619 13:10:03.617765 12795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:03.618016 12795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:03.618148 12795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:03.618311 12795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:03.620625 12795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:03.621138 12795 leader_election.cc:302] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b77a355a03d4d0b89591b6707924c44; no voters:
I20260619 13:10:03.621714 12795 leader_election.cc:288] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:10:03.621814 12800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:03.622574 12800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 LEADER]: Becoming Leader. State: Replica: 0b77a355a03d4d0b89591b6707924c44, State: Running, Role: LEADER
I20260619 13:10:03.623060 12800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:03.624006 12795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:03.625995 12802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b77a355a03d4d0b89591b6707924c44. Latest consensus state: current_term: 1 leader_uuid: "0b77a355a03d4d0b89591b6707924c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } } }
I20260619 13:10:03.626434 12802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:03.626300 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b77a355a03d4d0b89591b6707924c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } } }
I20260619 13:10:03.626566 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:03.630512 12807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:03.634707 12807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:03.635505 12795 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:10:03.655037 12807 catalog_manager.cc:1347] Generated new cluster ID: f666b0a2999c477d87d54482c80e8522
I20260619 13:10:03.655192 12807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:03.676276 12807 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:10:03.678349 12807 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:03.693037 12807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Generated new TSK 0
I20260619 13:10:03.694375 12807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:10:04.473212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:10:04 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:10:04.880146 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:04.880487 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:04.880656 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:04.888489 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:04.888638 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:04.888839 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:04.899533 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:04.901229 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:04.902894 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:04.921294 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:04.923161 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:04.923923 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:04.929315 12840 server_base.cc:1034] running on GCE node
I20260619 13:10:04.930125 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:04.932606 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:04.936033 12840 hybrid_clock.cc:648] HybridClock initialized: now 1781874604935965 us; error 49 us; skew 500 ppm
I20260619 13:10:04.936482 12840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:04 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:04.940248 12840 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:04.943024 12840 webserver.cc:460] Webserver started at http://127.0.97.1:43371/ using document root <none> and password file <none>
I20260619 13:10:04.943775 12840 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:04.943920 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:04.944293 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:10:04 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:04.960134 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "f0ba80d09dd84f66b57f8be6fbca8b31"
format_stamp: "Formatted at 2026-06-19 13:10:04 on dist-test-slave-q11v"
server_key: "97d293dba96ad474bd1bc27eeabc17f5"
server_key_iv: "47061b9418c81eb5f7ea16593492fb2c"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:04.961292 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f0ba80d09dd84f66b57f8be6fbca8b31"
format_stamp: "Formatted at 2026-06-19 13:10:04 on dist-test-slave-q11v"
server_key: "97d293dba96ad474bd1bc27eeabc17f5"
server_key_iv: "47061b9418c81eb5f7ea16593492fb2c"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:04.982820 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.012s sys 0.007s
I20260619 13:10:05.008467 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:05.010679 12840 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 13:10:05.010936 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "f0ba80d09dd84f66b57f8be6fbca8b31"
format_stamp: "Formatted at 2026-06-19 13:10:04 on dist-test-slave-q11v"
server_key: "97d293dba96ad474bd1bc27eeabc17f5"
server_key_iv: "47061b9418c81eb5f7ea16593492fb2c"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:05.011267 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:05.044695 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:05.049852 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:05.050171 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:05.051373 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:05.053503 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:05.053653 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 13:10:05.053872 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:05.053956 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:05.183629 12840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40315
I20260619 13:10:05.183647 12972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40315 every 8 connection(s)
I20260619 13:10:05.185361 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:05.185997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 12840
I20260619 13:10:05.186196 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:10:05.186957 388 external_mini_cluster.cc:1428] Setting key bdf8b9f18340fe5e9731e854c0963ddf
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 19 13:10:05 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:05.259749 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:05.206703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58465 (local address 127.0.97.62:36467)
0619 13:10:05.232553 (+ 25850us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:05.232560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:05.237701 (+ 5141us) server_negotiation.cc:407] Connection header received
0619 13:10:05.238030 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:05.238039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:05.238156 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:05.238286 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:05.240843 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:05.242143 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:05.245026 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:05.245885 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:05.246013 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:05.251262 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:05.251272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:05.251280 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:05.251329 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:05.255623 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:05.256794 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:05.256801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:05.256806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:05.256980 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:05.257574 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:05.257580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:05.257584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:05.258016 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:05.258134 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:05.259133 (+ 999us) server_negotiation.cc:299] Negotiation successful
0619 13:10:05.259370 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25698,"thread_start_us":25556,"threads_started":1}
I20260619 13:10:05.260921 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:05.205856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.1:58465)
0619 13:10:05.235062 (+ 29206us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:05.235128 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:05.237664 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:05.238814 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:05.238844 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:05.239556 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:05.240518 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:05.240552 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:05.242391 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:05.242399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:05.244778 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:05.244812 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:05.246416 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:05.246425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:05.246784 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:05.247698 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:05.247735 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:05.250940 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:05.255890 (+ 4950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:05.255903 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:05.255936 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:05.256630 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:05.257141 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:05.257147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:05.257152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:05.257429 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:05.258253 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:05.258265 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:05.258708 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 13:10:05.259032 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 13:10:05.259582 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28830,"thread_start_us":28590,"threads_started":1}
I20260619 13:10:05.262750 12973 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:05.263089 12973 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:05.263876 12973 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:05.266389 12760 ts_manager.cc:194] Registered new tserver with Master: f0ba80d09dd84f66b57f8be6fbca8b31 (127.0.97.1:40315)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:10:05.269309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:05.275862 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58465
Jun 19 13:10:05 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:10:05 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 13:10:05.700999 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:05.701527 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:05.701761 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:05.713441 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:05.713703 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:05.713960 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:05.730051 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:05.732208 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:05.734337 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:05.763440 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:05.764035 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:05.773983 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:05.776152 12979 server_base.cc:1034] running on GCE node
I20260619 13:10:05.777056 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:05.789944 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:05.797684 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781874605797520 us; error 158 us; skew 500 ppm
I20260619 13:10:05.798278 12979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:05 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:05.803349 12979 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:05.826093 12979 webserver.cc:460] Webserver started at http://127.0.97.2:41671/ using document root <none> and password file <none>
I20260619 13:10:05.827324 12979 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:05.827574 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:05.828047 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:10:05 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:05.851052 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "570516c1e038432d8d958c5a8b8d4934"
format_stamp: "Formatted at 2026-06-19 13:10:05 on dist-test-slave-q11v"
server_key: "2337556780607588e9ab72d25a3ba130"
server_key_iv: "edf2171059e9fc0dbcccbde89324e776"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:05.852094 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "570516c1e038432d8d958c5a8b8d4934"
format_stamp: "Formatted at 2026-06-19 13:10:05 on dist-test-slave-q11v"
server_key: "2337556780607588e9ab72d25a3ba130"
server_key_iv: "edf2171059e9fc0dbcccbde89324e776"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:05.859599 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260619 13:10:05.898660 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:05.900710 12979 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.001s
I20260619 13:10:05.900959 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "570516c1e038432d8d958c5a8b8d4934"
format_stamp: "Formatted at 2026-06-19 13:10:05 on dist-test-slave-q11v"
server_key: "2337556780607588e9ab72d25a3ba130"
server_key_iv: "edf2171059e9fc0dbcccbde89324e776"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:05.901207 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:05.928303 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:05.932538 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:05.932858 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:05.934471 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:05.935974 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:05.936079 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:05.936219 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:05.936302 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:05.985960 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39561
I20260619 13:10:05.985970 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39561 every 8 connection(s)
I20260619 13:10:05.987397 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:05.997576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 12979
I20260619 13:10:05.997790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:10:05.998411 388 external_mini_cluster.cc:1428] Setting key 091d7f4daa4a5fa2c38158f870118b1a
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:06.021554 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:05.991487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.2:33027)
0619 13:10:05.998239 (+ 6752us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:05.998537 (+ 298us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:06.001095 (+ 2558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:06.002271 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:06.002284 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:06.002930 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:06.003794 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:06.003819 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.005336 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.005341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:06.007461 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:06.007477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.008806 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.008814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:06.009134 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.009930 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:06.009961 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:06.012751 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:06.016772 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:06.016782 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:06.016802 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:06.017395 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:06.017866 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:06.017871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:06.017874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:06.018129 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:06.018830 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:06.018839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:06.019269 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 13:10:06.019569 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 13:10:06.020121 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6378,"thread_start_us":7482,"threads_started":1}
I20260619 13:10:06.023218 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:05.990632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33027 (local address 127.0.97.62:36467)
0619 13:10:05.998029 (+ 7397us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:05.998036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:06.000085 (+ 2049us) server_negotiation.cc:407] Connection header received
0619 13:10:06.001540 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:06.001560 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:06.001671 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:06.001785 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:06.004001 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.005143 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.007667 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.008286 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.008423 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.013023 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:06.013030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:06.013035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:06.013073 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:06.016598 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.017564 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.017570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.017573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.017738 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.018264 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.018269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.018273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.018627 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:06.018752 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:06.022627 (+ 3875us) server_negotiation.cc:299] Negotiation successful
0619 13:10:06.022871 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7249,"thread_start_us":7116,"threads_started":1}
I20260619 13:10:06.024991 13116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:06.025334 13116 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:06.026257 13116 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:06.028054 12760 ts_manager.cc:194] Registered new tserver with Master: 570516c1e038432d8d958c5a8b8d4934 (127.0.97.2:39561)
I20260619 13:10:06.029158 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33027
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:10:06.066849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:06.280289 12973 heartbeater.cc:503] Master 127.0.97.62:36467 was elected leader, sending a full tablet report...
W20260619 13:10:06.421496 13122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:06.421862 13122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:06.421970 13122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:06.429792 13122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:06.429908 13122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:06.430016 13122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:10:06.439777 13122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:06.441036 13122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:06.442261 13122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:06.455442 13127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:06.455322 13128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:06.455358 13130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:06.456502 13122 server_base.cc:1034] running on GCE node
I20260619 13:10:06.457123 13122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:06.458170 13122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:06.459414 13122 hybrid_clock.cc:648] HybridClock initialized: now 1781874606459364 us; error 42 us; skew 500 ppm
I20260619 13:10:06.459758 13122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:06.462842 13122 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:06.464304 13122 webserver.cc:460] Webserver started at http://127.0.97.3:38289/ using document root <none> and password file <none>
I20260619 13:10:06.465021 13122 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:06.465161 13122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:06.465463 13122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:06.482276 13122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "badf3c3932a04ed281783959d95529e7"
format_stamp: "Formatted at 2026-06-19 13:10:06 on dist-test-slave-q11v"
server_key: "03db8f566180d8d3bde237c49d4d105f"
server_key_iv: "c5846af9819e959262fb1b05149506a0"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:06.483116 13122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "badf3c3932a04ed281783959d95529e7"
format_stamp: "Formatted at 2026-06-19 13:10:06 on dist-test-slave-q11v"
server_key: "03db8f566180d8d3bde237c49d4d105f"
server_key_iv: "c5846af9819e959262fb1b05149506a0"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:06.488792 13122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 13:10:06.504926 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:06.506603 13122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:06.506786 13122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "badf3c3932a04ed281783959d95529e7"
format_stamp: "Formatted at 2026-06-19 13:10:06 on dist-test-slave-q11v"
server_key: "03db8f566180d8d3bde237c49d4d105f"
server_key_iv: "c5846af9819e959262fb1b05149506a0"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:06.507014 13122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:06.522328 13122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:06.526448 13122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:06.526747 13122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:06.527683 13122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:06.529085 13122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:06.529157 13122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:06.529332 13122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:06.529384 13122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:06.560945 13122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35813
I20260619 13:10:06.561034 13249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35813 every 8 connection(s)
I20260619 13:10:06.562309 13122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:06.566840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13122
I20260619 13:10:06.567016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:10:06.567659 388 external_mini_cluster.cc:1428] Setting key 29f1a57c4baaf2f997c81deeb7673a75
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:06.584543 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:06.566398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55739 (local address 127.0.97.62:36467)
0619 13:10:06.566956 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:06.566965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:06.567912 (+ 947us) server_negotiation.cc:407] Connection header received
0619 13:10:06.570256 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:06.570262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:06.570383 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:06.570494 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:06.572062 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.572908 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.574442 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.574859 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.574953 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.578270 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:06.578274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:06.578277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:06.578310 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:06.580968 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.581885 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.581890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.581893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.582087 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.582602 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.582609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.582611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.582931 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:06.583126 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:06.583945 (+ 819us) server_negotiation.cc:299] Negotiation successful
0619 13:10:06.584179 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":264,"threads_started":1}
I20260619 13:10:06.585498 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:06.566790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.3:55739)
0619 13:10:06.567725 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:06.567780 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:06.569991 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:06.570774 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:06.570786 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:06.571262 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:06.571902 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:06.571920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.573023 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.573026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:06.574327 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:06.574340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.575072 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.575077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:06.575281 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.575942 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:06.575962 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:06.578068 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:06.581116 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:06.581125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:06.581142 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:06.581761 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:06.582212 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:06.582217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:06.582220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:06.582457 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:06.583184 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:06.583191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:06.583585 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 13:10:06.583847 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 13:10:06.584370 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":400,"threads_started":1}
I20260619 13:10:06.590898 13250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:06.591214 13250 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:06.591935 13250 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:06.593343 12760 ts_manager.cc:194] Registered new tserver with Master: badf3c3932a04ed281783959d95529e7 (127.0.97.3:35813)
I20260619 13:10:06.594205 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55739
I20260619 13:10:06.595595 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:10:06.597218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 12840
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:10:06.613489 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:06.598069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.0.97.1:40315)
0619 13:10:06.598646 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:06.598653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:06.598677 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:10:06.598846 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:06.598853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:06.599053 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:06.599175 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:06.600527 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.601495 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.602989 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.603512 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.603607 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.607870 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:06.607881 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:06.607896 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:06.607951 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:06.610792 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.611461 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.611470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.611473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.611614 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.611951 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.611954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.611956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.612372 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:06.612650 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:06.612868 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 13:10:06.613081 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":282,"threads_started":1}
I20260619 13:10:06.614773 12968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:06.698333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 12840
I20260619 13:10:06.714406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 12979
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:10:06.730477 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:06.715290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.0.97.2:39561)
0619 13:10:06.715878 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:06.715885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:06.715911 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:10:06.715972 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:06.715979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:06.716213 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:06.716345 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:06.717668 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.718687 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.719972 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.720535 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.720632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.725018 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:06.725029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:06.725044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:06.725079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:06.727852 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.728575 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.728582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.728584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.728718 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.729054 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.729057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.729058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.729409 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:06.729693 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:06.730002 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 13:10:06.730198 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260619 13:10:06.731765 13111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:06.813858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 12979
I20260619 13:10:06.834007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13122
Jun 19 13:10:06 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:10:06.851086 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:06.835004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.97.3:35813)
0619 13:10:06.835636 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:06.835643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:06.835673 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:10:06.835730 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:06.835736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:06.835969 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:06.836098 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:06.837485 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.838539 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.839895 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:06.840439 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:06.840533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:06.845424 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:06.845436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:06.845452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:06.845492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:06.848367 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.849024 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.849031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.849034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.849169 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:06.849470 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:06.849473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:06.849475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:06.849893 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:06.850157 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:06.850479 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 13:10:06.850701 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
I20260619 13:10:06.852366 13245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:06.935365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13122
I20260619 13:10:06.950891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 12728
I20260619 13:10:06.952088 12790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:07.057504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 12728
I20260619 13:10:07.084406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36467
--webserver_interface=127.0.97.62
--webserver_port=33961
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36467
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:07.417179 13262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:07.417470 13262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:07.417703 13262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:07.425050 13262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:07.425171 13262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:07.425262 13262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:07.425346 13262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:07.425431 13262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:07.436053 13262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36467
--ranger_config_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:07.437455 13262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:07.438850 13262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:07.450984 13267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:07.451021 13268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:07.452327 13270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:07.452795 13262 server_base.cc:1034] running on GCE node
I20260619 13:10:07.453682 13262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:07.455225 13262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:07.456439 13262 hybrid_clock.cc:648] HybridClock initialized: now 1781874607456390 us; error 37 us; skew 500 ppm
I20260619 13:10:07.456861 13262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:07 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:07.460175 13262 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:07.461787 13262 webserver.cc:460] Webserver started at http://127.0.97.62:33961/ using document root <none> and password file <none>
I20260619 13:10:07.462452 13262 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:07.462586 13262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:10:07 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:07.513948 13262 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:10:07.517275 13278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:07.518815 13262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:07.519042 13262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "0b77a355a03d4d0b89591b6707924c44"
format_stamp: "Formatted at 2026-06-19 13:10:03 on dist-test-slave-q11v"
server_key: "8a140398d6be90031ff40b473afc03d4"
server_key_iv: "2498df2e8c5d4cb978ddd414bfd1094b"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:07.520035 13262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:07.545272 13262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:07.549345 13262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:07.549715 13262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:07.570472 13262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36467
I20260619 13:10:07.570500 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36467 every 8 connection(s)
I20260619 13:10:07.572031 13262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:07.579268 13330 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:10:07.580250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13262
I20260619 13:10:07.581065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40315
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36467
--webserver_port=43371
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:07.582878 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Bootstrap starting.
I20260619 13:10:07.601517 13330 log.cc:824] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:07.610508 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:10:07.611068 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Bootstrap complete.
I20260619 13:10:07.620162 13330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:07.621039 13330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b77a355a03d4d0b89591b6707924c44, State: Initialized, Role: FOLLOWER
I20260619 13:10:07.621678 13330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:07.621917 13330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:07.622038 13330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:07.622216 13330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:10:07.626448 13330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:07.626950 13330 leader_election.cc:302] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b77a355a03d4d0b89591b6707924c44; no voters:
I20260619 13:10:07.627596 13330 leader_election.cc:288] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:10:07.627737 13334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:10:07.628928 13334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [term 2 LEADER]: Becoming Leader. State: Replica: 0b77a355a03d4d0b89591b6707924c44, State: Running, Role: LEADER
I20260619 13:10:07.629563 13334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } }
I20260619 13:10:07.629941 13330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:07.632314 13335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b77a355a03d4d0b89591b6707924c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } } }
I20260619 13:10:07.632782 13335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:07.636597 13336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b77a355a03d4d0b89591b6707924c44. Latest consensus state: current_term: 2 leader_uuid: "0b77a355a03d4d0b89591b6707924c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b77a355a03d4d0b89591b6707924c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36467 } } }
I20260619 13:10:07.637038 13336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:07.638794 13340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:07.645641 13340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:07.646498 13330 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:10:07.647074 13340 catalog_manager.cc:1259] Loaded cluster ID: f666b0a2999c477d87d54482c80e8522
I20260619 13:10:07.647239 13340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:07.651145 13340 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:07.652693 13340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Loaded TSK: 0
I20260619 13:10:07.654452 13340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:10:08.018078 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:08.018498 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:08.018708 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:08.031001 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:08.031226 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:08.031479 13332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:08.048182 13332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43371
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:08.050001 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:08.051748 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:08.087616 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:08.093513 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:08.094398 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:08.095338 13332 server_base.cc:1034] running on GCE node
I20260619 13:10:08.096359 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:08.097898 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:08.099246 13332 hybrid_clock.cc:648] HybridClock initialized: now 1781874608099130 us; error 101 us; skew 500 ppm
I20260619 13:10:08.099660 13332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:08 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:08.105254 13332 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:08.111480 13332 webserver.cc:460] Webserver started at http://127.0.97.1:43371/ using document root <none> and password file <none>
I20260619 13:10:08.112763 13332 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:08.113116 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:10:08 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:08.148142 13332 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260619 13:10:08.161545 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.164244 13332 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.007s sys 0.005s
I20260619 13:10:08.164572 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "f0ba80d09dd84f66b57f8be6fbca8b31"
format_stamp: "Formatted at 2026-06-19 13:10:04 on dist-test-slave-q11v"
server_key: "97d293dba96ad474bd1bc27eeabc17f5"
server_key_iv: "47061b9418c81eb5f7ea16593492fb2c"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:08.166210 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:08.233750 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:08.239451 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:08.239825 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:08.253053 13332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:08.266533 13332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:08.266741 13332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.266916 13332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:08.267038 13332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.343832 13332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40315
I20260619 13:10:08.343916 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40315 every 8 connection(s)
I20260619 13:10:08.345264 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:08.349227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13332
I20260619 13:10:08.349954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39561
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36467
--webserver_port=41671
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:10:08 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:08.372869 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:08.348267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.1:35007)
0619 13:10:08.349295 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:08.349360 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:08.352784 (+ 3424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:08.355191 (+ 2407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:08.355204 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:08.355762 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:08.356325 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:08.356343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:08.358657 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:08.358667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:08.359896 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:08.359909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:08.360969 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:08.360976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:08.361236 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:08.361986 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:08.362009 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:08.364078 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:08.368087 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:08.368098 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:08.368115 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:08.368571 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:08.369098 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:08.369103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:08.369106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:08.369314 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:08.370160 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:08.370168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:08.370531 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 13:10:08.370817 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 13:10:08.371401 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":368,"threads_started":1}
I20260619 13:10:08.372848 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:08.348264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35007 (local address 127.0.97.62:36467)
0619 13:10:08.353831 (+ 5567us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:08.353851 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:08.353903 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:10:08.354263 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:08.354278 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:08.354650 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:08.355064 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:08.356555 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:08.358479 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:08.360045 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:08.360727 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:08.360932 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:08.364417 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:08.364426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:08.364446 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:08.364492 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:08.367906 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:08.368727 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:08.368735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:08.368748 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:08.368981 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:08.369428 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:08.369434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:08.369437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:08.370017 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:08.370231 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:08.371030 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 13:10:08.371401 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5287,"thread_start_us":5081,"threads_started":1}
I20260619 13:10:08.376504 13492 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:08.376863 13492 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:08.377835 13492 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
W20260619 13:10:08.381036 13492 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36467 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:10:08.381265 13492 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:08.381558 13492 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:08.382992 13492 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:08.383304 13492 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
W20260619 13:10:08.384482 13492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:10:08.783553 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:08.783910 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:08.784123 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:08.793520 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:08.793833 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:08.794061 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:08.807077 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41671
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:08.808552 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:08.809990 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:08.823487 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:08.823467 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:08.823411 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:08.825244 13496 server_base.cc:1034] running on GCE node
I20260619 13:10:08.826064 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:08.827334 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:08.828606 13496 hybrid_clock.cc:648] HybridClock initialized: now 1781874608828514 us; error 53 us; skew 500 ppm
I20260619 13:10:08.829041 13496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:08 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:08.838488 13496 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:08.840234 13496 webserver.cc:460] Webserver started at http://127.0.97.2:41671/ using document root <none> and password file <none>
I20260619 13:10:08.840947 13496 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:08.841104 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:10:08 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:08.865746 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 13:10:08.869524 13521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.871452 13496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:10:08.871644 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "570516c1e038432d8d958c5a8b8d4934"
format_stamp: "Formatted at 2026-06-19 13:10:05 on dist-test-slave-q11v"
server_key: "2337556780607588e9ab72d25a3ba130"
server_key_iv: "edf2171059e9fc0dbcccbde89324e776"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:08.873044 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:08.900851 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:08.906351 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:08.906667 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:08.917970 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:08.924948 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:08.925092 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.925220 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:08.925298 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:08.993111 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39561
I20260619 13:10:08.993306 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39561 every 8 connection(s)
I20260619 13:10:08.994956 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:09.001933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13496
I20260619 13:10:09.002703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35813
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36467
--webserver_port=38289
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:09.031437 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:08.999240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50757 (local address 127.0.97.62:36467)
0619 13:10:09.004653 (+ 5413us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.004660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.004682 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:10:09.004749 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.004755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.004871 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.004995 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:09.009337 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.010962 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.016592 (+ 5630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.017314 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.017437 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.022454 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:09.022463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:09.022467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:09.022518 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:09.027025 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.028121 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.028128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.028131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.028309 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.029008 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.029015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.029020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.029402 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:09.029530 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:09.030800 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0619 13:10:09.031059 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5273,"thread_start_us":6809,"threads_started":1}
I20260619 13:10:09.034008 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:08.998578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.2:50757)
0619 13:10:09.000947 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:09.000999 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:09.003453 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:09.007491 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:09.007506 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:09.008134 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:09.009073 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.009104 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.013831 (+ 4727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.013837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:09.016365 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.016386 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.017860 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.017867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:09.018199 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.019036 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:09.019068 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:09.022096 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:09.027240 (+ 5144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:09.027250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:09.027272 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:09.027957 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:09.028474 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:09.028479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:09.028483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:09.028864 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:09.029670 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:09.029679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:09.030271 (+ 592us) client_negotiation.cc:769] Sending connection context
0619 13:10:09.031776 (+ 1505us) client_negotiation.cc:240] Negotiation successful
0619 13:10:09.032411 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2058,"thread_start_us":1848,"threads_started":1}
I20260619 13:10:09.036075 13636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:09.036422 13636 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:09.037168 13636 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:09.039317 13295 ts_manager.cc:194] Registered new tserver with Master: 570516c1e038432d8d958c5a8b8d4934 (127.0.97.2:39561)
I20260619 13:10:09.041613 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50757
I20260619 13:10:09.384645 13492 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:09.385116 13492 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:09.386390 13295 ts_manager.cc:194] Registered new tserver with Master: f0ba80d09dd84f66b57f8be6fbca8b31 (127.0.97.1:40315)
I20260619 13:10:09.387547 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35007
W20260619 13:10:09.458140 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:09.458649 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:09.458841 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:09.470418 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:09.470644 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:09.470834 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:09.486503 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38289
--tserver_master_addrs=127.0.97.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:09.488302 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:09.490123 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:09.534602 13646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:09.546428 13647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:09.547286 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:09.548461 13640 server_base.cc:1034] running on GCE node
I20260619 13:10:09.549583 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:09.555855 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:09.557106 13640 hybrid_clock.cc:648] HybridClock initialized: now 1781874609557043 us; error 84 us; skew 500 ppm
I20260619 13:10:09.557526 13640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:09.561902 13640 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:09.589651 13640 webserver.cc:460] Webserver started at http://127.0.97.3:38289/ using document root <none> and password file <none>
I20260619 13:10:09.590831 13640 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:09.591019 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:09.650009 13640 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.008s
I20260619 13:10:09.654421 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:09.656436 13640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:10:09.656629 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "badf3c3932a04ed281783959d95529e7"
format_stamp: "Formatted at 2026-06-19 13:10:06 on dist-test-slave-q11v"
server_key: "03db8f566180d8d3bde237c49d4d105f"
server_key_iv: "c5846af9819e959262fb1b05149506a0"
server_key_version: "kuduclusterkey@0"
I20260619 13:10:09.658077 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:09.676462 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:09.682264 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:09.682605 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:09.683889 13640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:09.685817 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:09.685935 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:09.686049 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:09.686151 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:09.746599 13640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35813
I20260619 13:10:09.746644 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35813 every 8 connection(s)
I20260619 13:10:09.748891 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:09.749449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13640
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:09.821884 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.770489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40109 (local address 127.0.97.62:36467)
0619 13:10:09.795097 (+ 24608us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.795105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.799419 (+ 4314us) server_negotiation.cc:407] Connection header received
0619 13:10:09.799518 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.799524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.799658 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.799817 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:09.803496 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.805830 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.808102 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.808666 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.808774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.813108 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:09.813114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:09.813119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:09.813159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:09.817514 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.818554 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.818559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.818562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.818713 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.819334 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.819340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.819343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.819678 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:09.819817 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:09.821179 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0619 13:10:09.821430 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24414,"thread_start_us":24629,"threads_started":1}
I20260619 13:10:09.824170 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.769746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36467 (local address 127.0.97.3:40109)
0619 13:10:09.795883 (+ 26137us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:09.795947 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:09.798840 (+ 2893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:09.801773 (+ 2933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:09.801788 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:09.802458 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:09.803279 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.803304 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.805992 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.805997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:09.807912 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.807927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.809087 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.809094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:09.809378 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.810256 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:09.810287 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:09.812839 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:09.817728 (+ 4889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:09.817738 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:09.817760 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:09.818374 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:09.818852 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:09.818857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:09.818861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:09.819131 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:09.819890 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:09.819898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:09.820360 (+ 462us) client_negotiation.cc:769] Sending connection context
0619 13:10:09.820668 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 13:10:09.822612 (+ 1944us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25744,"thread_start_us":31047,"threads_started":1}
I20260619 13:10:09.826092 13770 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36467
I20260619 13:10:09.826498 13770 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:09.827446 13770 heartbeater.cc:511] Master 127.0.97.62:36467 requested a full tablet report, sending...
I20260619 13:10:09.829277 13295 ts_manager.cc:194] Registered new tserver with Master: badf3c3932a04ed281783959d95529e7 (127.0.97.3:35813)
I20260619 13:10:09.830793 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40109
I20260619 13:10:09.832425 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.750668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.0.97.62:36467)
0619 13:10:09.764435 (+ 13767us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.764444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.800704 (+ 36260us) server_negotiation.cc:407] Connection header received
0619 13:10:09.800936 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.800945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.801040 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.801165 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:09.804392 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.805445 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.814354 (+ 8909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.814961 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.815082 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.825578 (+ 10496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:09.825584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:09.825589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:09.825648 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:09.828853 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.829899 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.829904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.829907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.830126 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.830607 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.830611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.830618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.830931 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:09.831208 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:09.831856 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 13:10:09.832082 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13587,"thread_start_us":13774,"threads_started":1}
I20260619 13:10:09.838510 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:10:09 dist-test-slave-q11v krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:10:09.884028 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.846067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.0.97.62:36467)
0619 13:10:09.846398 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.846409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.863184 (+ 16775us) server_negotiation.cc:407] Connection header received
0619 13:10:09.863838 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.863846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.863956 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.864118 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:09.866469 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.867652 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.870084 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.870817 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.870950 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:09.875285 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:09.875296 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:09.875305 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:09.875360 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:09.879437 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.880943 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.880950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.880954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.881149 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:09.881794 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:09.881800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:09.881804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:09.882248 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:09.882535 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:09.883261 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 13:10:09.883565 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 13:10:09.892973 13295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:10:09.905907 13295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:10:10.015417 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.942321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40315 (local address 127.0.0.1:36404)
0619 13:10:09.980263 (+ 37942us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:09.980335 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:09.980618 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:10.004136 (+ 23518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:10.004141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:10.004170 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:10.004587 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.004603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.007355 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.007362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.010475 (+ 3113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.010496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.014124 (+ 3628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.014133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.014419 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.014509 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 13:10:10.014649 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 13:10:10.014946 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37596,"thread_start_us":60803,"threads_started":1}
I20260619 13:10:10.017491 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.945067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35813 (local address 127.0.0.1:51292)
0619 13:10:09.986995 (+ 41928us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:09.987031 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:09.987278 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:09.989334 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:09.989349 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:09.989405 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:09.990251 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.990274 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.993691 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:09.993701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:09.996632 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.996655 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.016704 (+ 20049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.016711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.016892 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.016925 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:10:10.016994 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 13:10:10.017138 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41706,"thread_start_us":42361,"threads_started":1}
I20260619 13:10:10.019121 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.942618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.0.97.1:40315)
0619 13:10:10.003132 (+ 60514us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.003143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.003181 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:10:10.003316 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.003327 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.003638 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.003873 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:10.005396 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.007148 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.011936 (+ 4788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.013820 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.018097 (+ 4277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.018399 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.018500 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.018700 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60288,"thread_start_us":61461,"threads_started":1}
I20260619 13:10:10.019157 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.945877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.0.97.3:35813)
0619 13:10:09.980008 (+ 34131us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.980021 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.987825 (+ 7804us) server_negotiation.cc:407] Connection header received
0619 13:10:09.988005 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.988020 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.988470 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.988782 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:09.990622 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.992816 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.004421 (+ 11605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.013838 (+ 9417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.018252 (+ 4414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.018520 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.018596 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.018766 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33873,"thread_start_us":34162,"threads_started":1}
I20260619 13:10:10.023502 13705 tablet_service.cc:1467] Processing CreateTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a (DEFAULT_TABLE table=test-table [id=ef2b03823a8040359a9e7e07ecb224cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:10.025143 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad8ba54d705403e9bfbff65b4b47e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:10.028851 13427 tablet_service.cc:1467] Processing CreateTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a (DEFAULT_TABLE table=test-table [id=ef2b03823a8040359a9e7e07ecb224cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:10.030714 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad8ba54d705403e9bfbff65b4b47e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:10.046495 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.946260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.97.2:39561)
0619 13:10:09.981346 (+ 35086us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:09.981358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:09.996091 (+ 14733us) server_negotiation.cc:407] Connection header received
0619 13:10:09.996214 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:09.996223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:09.996531 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:09.996786 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:09.997877 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:09.999782 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.014081 (+ 14299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.015606 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.015720 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.015968 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.045836 (+ 29868us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.046077 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34893,"thread_start_us":35857,"threads_started":1}
I20260619 13:10:10.047552 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:09.945623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39561 (local address 127.0.0.1:46842)
0619 13:10:09.993444 (+ 47821us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:09.993470 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:09.993775 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:09.996829 (+ 3054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:09.996834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:09.996871 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:09.997279 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:09.997293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.000041 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.000047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.002397 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.002416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.016187 (+ 13771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.016194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.016386 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.016419 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:10:10.047007 (+ 30588us) client_negotiation.cc:240] Negotiation successful
0619 13:10:10.047187 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47663,"thread_start_us":35761,"threads_started":1}
I20260619 13:10:10.051254 13636 heartbeater.cc:503] Master 127.0.97.62:36467 was elected leader, sending a full tablet report...
I20260619 13:10:10.052335 13571 tablet_service.cc:1467] Processing CreateTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a (DEFAULT_TABLE table=test-table [id=ef2b03823a8040359a9e7e07ecb224cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:10.054106 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad8ba54d705403e9bfbff65b4b47e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:10.080071 13792 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Bootstrap starting.
I20260619 13:10:10.084403 13792 tablet_bootstrap.cc:654] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:10.085353 13794 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Bootstrap starting.
I20260619 13:10:10.086336 13793 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Bootstrap starting.
I20260619 13:10:10.090202 13793 tablet_bootstrap.cc:654] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:10.090557 13794 tablet_bootstrap.cc:654] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:10.091370 13792 log.cc:824] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:10.091475 13793 log.cc:824] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:10.092898 13794 log.cc:824] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:10.099392 13794 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: No bootstrap required, opened a new log
I20260619 13:10:10.099817 13794 ts_tablet_manager.cc:1397] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260619 13:10:10.102317 13793 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: No bootstrap required, opened a new log
I20260619 13:10:10.102711 13793 ts_tablet_manager.cc:1397] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260619 13:10:10.109932 13792 tablet_bootstrap.cc:492] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: No bootstrap required, opened a new log
I20260619 13:10:10.110353 13792 ts_tablet_manager.cc:1397] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.003s
I20260619 13:10:10.113799 13793 raft_consensus.cc:348] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.114455 13793 raft_consensus.cc:374] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:10.114601 13793 raft_consensus.cc:729] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badf3c3932a04ed281783959d95529e7, State: Initialized, Role: FOLLOWER
I20260619 13:10:10.115478 13793 consensus_queue.cc:260] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.119587 13794 raft_consensus.cc:348] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.120168 13794 raft_consensus.cc:374] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:10.120318 13794 raft_consensus.cc:729] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ba80d09dd84f66b57f8be6fbca8b31, State: Initialized, Role: FOLLOWER
I20260619 13:10:10.120859 13770 heartbeater.cc:503] Master 127.0.97.62:36467 was elected leader, sending a full tablet report...
I20260619 13:10:10.121294 13794 consensus_queue.cc:260] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.121975 13793 ts_tablet_manager.cc:1428] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260619 13:10:10.122402 13792 raft_consensus.cc:348] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.122911 13792 raft_consensus.cc:374] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:10.123054 13792 raft_consensus.cc:729] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570516c1e038432d8d958c5a8b8d4934, State: Initialized, Role: FOLLOWER
I20260619 13:10:10.123741 13792 consensus_queue.cc:260] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.129977 13794 ts_tablet_manager.cc:1428] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Time spent starting tablet: real 0.030s user 0.020s sys 0.000s
I20260619 13:10:10.130439 13492 heartbeater.cc:503] Master 127.0.97.62:36467 was elected leader, sending a full tablet report...
I20260619 13:10:10.130998 13792 ts_tablet_manager.cc:1428] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
W20260619 13:10:10.257908 13637 tablet.cc:2369] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:10.263006 13802 raft_consensus.cc:482] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:10:10.263317 13802 raft_consensus.cc:504] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.265267 13802 leader_election.cc:288] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0ba80d09dd84f66b57f8be6fbca8b31 (127.0.97.1:40315), badf3c3932a04ed281783959d95529e7 (127.0.97.3:35813)
I20260619 13:10:10.272260 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.265951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33833 (local address 127.0.97.3:35813)
0619 13:10:10.266171 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.266180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.266916 (+ 736us) server_negotiation.cc:407] Connection header received
0619 13:10:10.267137 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.267142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.267197 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.267292 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:10.267911 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.268646 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.270557 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.271303 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.271451 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.271556 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.271842 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.271967 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:10:10.272482 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.265550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35813 (local address 127.0.97.2:33833)
0619 13:10:10.266815 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:10.266830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:10.266996 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:10.267307 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:10.267311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:10.267342 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:10.267752 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.267761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.268781 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.268788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.270403 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.270415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.271503 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.271509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.271669 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.271693 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:10:10.271818 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 13:10:10.272006 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":25,"spinlock_wait_cycles":14464,"thread_start_us":684,"threads_started":1}
I20260619 13:10:10.272897 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.265649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44545 (local address 127.0.97.1:40315)
0619 13:10:10.265909 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.265917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.266899 (+ 982us) server_negotiation.cc:407] Connection header received
0619 13:10:10.267143 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.267148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.267222 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.267326 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:10.267994 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.268793 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.270923 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.271768 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.271947 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.272068 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.272478 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.272619 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:10:10.273742 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.265516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40315 (local address 127.0.97.2:44545)
0619 13:10:10.266772 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:10.266789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:10.266987 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:10.267343 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:10.267348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:10.267373 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:10.267841 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.267851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.268954 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.268964 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.270767 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:10.270779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.272041 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.272049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:10.272242 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.272267 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:10:10.273095 (+ 828us) client_negotiation.cc:240] Negotiation successful
0619 13:10:10.273275 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":1,"spinlock_wait_cycles":3456,"thread_start_us":963,"threads_started":1}
I20260619 13:10:10.273777 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad8ba54d705403e9bfbff65b4b47e6a" candidate_uuid: "570516c1e038432d8d958c5a8b8d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf3c3932a04ed281783959d95529e7" is_pre_election: true
I20260619 13:10:10.274370 13725 raft_consensus.cc:2454] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 570516c1e038432d8d958c5a8b8d4934 in term 0.
I20260619 13:10:10.274762 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad8ba54d705403e9bfbff65b4b47e6a" candidate_uuid: "570516c1e038432d8d958c5a8b8d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" is_pre_election: true
I20260619 13:10:10.275036 13524 leader_election.cc:302] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 570516c1e038432d8d958c5a8b8d4934, badf3c3932a04ed281783959d95529e7; no voters:
I20260619 13:10:10.275241 13447 raft_consensus.cc:2454] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 570516c1e038432d8d958c5a8b8d4934 in term 0.
I20260619 13:10:10.275455 13802 raft_consensus.cc:2781] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:10:10.275621 13802 raft_consensus.cc:482] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:10:10.275738 13802 raft_consensus.cc:3037] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:10.278084 13802 raft_consensus.cc:504] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.278729 13802 leader_election.cc:288] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [CANDIDATE]: Term 1 election: Requested vote from peers f0ba80d09dd84f66b57f8be6fbca8b31 (127.0.97.1:40315), badf3c3932a04ed281783959d95529e7 (127.0.97.3:35813)
I20260619 13:10:10.279201 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad8ba54d705403e9bfbff65b4b47e6a" candidate_uuid: "570516c1e038432d8d958c5a8b8d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31"
I20260619 13:10:10.279227 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad8ba54d705403e9bfbff65b4b47e6a" candidate_uuid: "570516c1e038432d8d958c5a8b8d4934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf3c3932a04ed281783959d95529e7"
I20260619 13:10:10.279398 13447 raft_consensus.cc:3037] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:10.279433 13725 raft_consensus.cc:3037] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:10.281549 13447 raft_consensus.cc:2454] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 570516c1e038432d8d958c5a8b8d4934 in term 1.
I20260619 13:10:10.281952 13725 raft_consensus.cc:2454] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 570516c1e038432d8d958c5a8b8d4934 in term 1.
I20260619 13:10:10.282019 13523 leader_election.cc:302] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 570516c1e038432d8d958c5a8b8d4934, f0ba80d09dd84f66b57f8be6fbca8b31; no voters:
I20260619 13:10:10.282415 13802 raft_consensus.cc:2781] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:10.282894 13802 raft_consensus.cc:686] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 1 LEADER]: Becoming Leader. State: Replica: 570516c1e038432d8d958c5a8b8d4934, State: Running, Role: LEADER
I20260619 13:10:10.283393 13802 consensus_queue.cc:237] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } }
I20260619 13:10:10.288273 13293 catalog_manager.cc:5495] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 570516c1e038432d8d958c5a8b8d4934 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "570516c1e038432d8d958c5a8b8d4934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "570516c1e038432d8d958c5a8b8d4934" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39561 } health_report { overall_health: HEALTHY } } }
W20260619 13:10:10.294353 13775 tablet.cc:2369] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:10.317812 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.312363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.0.97.2:39561)
0619 13:10:10.312585 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.312591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.312620 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:10:10.312863 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.312867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.312943 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.313097 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:10:10.313515 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.314332 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.315620 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.316014 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.316148 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.316628 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:10:10.317228 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:10:10.317399 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.317448 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.317524 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:10:10.326268 13725 raft_consensus.cc:1261] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 1 FOLLOWER]: Refusing update from remote peer 570516c1e038432d8d958c5a8b8d4934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:10.326267 13447 raft_consensus.cc:1261] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 1 FOLLOWER]: Refusing update from remote peer 570516c1e038432d8d958c5a8b8d4934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:10.327325 13810 consensus_queue.cc:1035] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ba80d09dd84f66b57f8be6fbca8b31" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:10.327693 13802 consensus_queue.cc:1035] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badf3c3932a04ed281783959d95529e7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:10.340250 13814 mvcc.cc:205] Tried to move back new op lower bound from 7298558403885191168 to 7298558403724357632. Current Snapshot: MvccSnapshot[applied={T|T < 7298558403885191168}]
W20260619 13:10:10.348196 13493 tablet.cc:2369] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:10.352315 13815 mvcc.cc:205] Tried to move back new op lower bound from 7298558403885191168 to 7298558403724357632. Current Snapshot: MvccSnapshot[applied={T|T < 7298558403885191168}]
I20260619 13:10:10.363608 13294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:10.363907 13294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:10.367879 13294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b77a355a03d4d0b89591b6707924c44: Sending DeleteTablet for 3 replicas of tablet aad8ba54d705403e9bfbff65b4b47e6a
I20260619 13:10:10.368995 13705 tablet_service.cc:1514] Processing DeleteTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51292
I20260619 13:10:10.369004 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36404
I20260619 13:10:10.369980 13823 tablet_replica.cc:330] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: stopping tablet replica
I20260619 13:10:10.370519 13822 tablet_replica.cc:330] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: stopping tablet replica
I20260619 13:10:10.370721 13571 tablet_service.cc:1514] Processing DeleteTablet for tablet aad8ba54d705403e9bfbff65b4b47e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46842
I20260619 13:10:10.371012 13822 raft_consensus.cc:2229] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:10.371580 13822 raft_consensus.cc:2258] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:10.373209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13332
I20260619 13:10:10.374382 13822 ts_tablet_manager.cc:1905] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:10.378710 13823 raft_consensus.cc:2229] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:10.379518 13823 raft_consensus.cc:2258] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:10.381099 13824 tablet_replica.cc:330] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: stopping tablet replica
I20260619 13:10:10.381691 13824 raft_consensus.cc:2229] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:10:10.382474 13824 raft_consensus.cc:2258] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:10.383244 13823 ts_tablet_manager.cc:1905] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:10.384555 13824 ts_tablet_manager.cc:1905] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:10.390503 13823 ts_tablet_manager.cc:1918] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:10.392205 13824 ts_tablet_manager.cc:1918] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:10.392460 13824 log.cc:1196] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/aad8ba54d705403e9bfbff65b4b47e6a
I20260619 13:10:10.393044 13824 ts_tablet_manager.cc:1939] T aad8ba54d705403e9bfbff65b4b47e6a P 570516c1e038432d8d958c5a8b8d4934: Deleting consensus metadata
I20260619 13:10:10.393702 13822 ts_tablet_manager.cc:1918] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:10.393874 13822 log.cc:1196] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/aad8ba54d705403e9bfbff65b4b47e6a
I20260619 13:10:10.394434 13822 ts_tablet_manager.cc:1939] T aad8ba54d705403e9bfbff65b4b47e6a P badf3c3932a04ed281783959d95529e7: Deleting consensus metadata
I20260619 13:10:10.395574 13282 catalog_manager.cc:4841] TS 570516c1e038432d8d958c5a8b8d4934 (127.0.97.2:39561): tablet aad8ba54d705403e9bfbff65b4b47e6a (table test-table [id=ef2b03823a8040359a9e7e07ecb224cf]) successfully deleted
I20260619 13:10:10.396210 13823 log.cc:1196] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/aad8ba54d705403e9bfbff65b4b47e6a
I20260619 13:10:10.396279 13281 catalog_manager.cc:4841] TS badf3c3932a04ed281783959d95529e7 (127.0.97.3:35813): tablet aad8ba54d705403e9bfbff65b4b47e6a (table test-table [id=ef2b03823a8040359a9e7e07ecb224cf]) successfully deleted
I20260619 13:10:10.396786 13823 ts_tablet_manager.cc:1939] T aad8ba54d705403e9bfbff65b4b47e6a P f0ba80d09dd84f66b57f8be6fbca8b31: Deleting consensus metadata
I20260619 13:10:10.398456 13280 catalog_manager.cc:4841] TS f0ba80d09dd84f66b57f8be6fbca8b31 (127.0.97.1:40315): tablet aad8ba54d705403e9bfbff65b4b47e6a (table test-table [id=ef2b03823a8040359a9e7e07ecb224cf]) successfully deleted
I20260619 13:10:10.403364 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.374623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.1:40315)
0619 13:10:10.374806 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.374812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.385428 (+ 10616us) server_negotiation.cc:407] Connection header received
0619 13:10:10.385943 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.385950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.386054 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.386147 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:10.388561 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.389764 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.391730 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.392365 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.392478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.396259 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:10.396274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:10.396292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:10.396338 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:10.400404 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.401074 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.401084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.401087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.401247 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.401704 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.401711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.401713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.402171 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:10.402482 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.402750 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.402979 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:10:10.404028 13487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:10.530835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13332
I20260619 13:10:10.547676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13496
I20260619 13:10:10.563336 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.548514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.97.2:39561)
0619 13:10:10.548697 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.548703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.548729 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:10:10.548912 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.548918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.549007 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.549122 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:10.550462 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.551639 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.552934 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.553759 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.553899 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.556376 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:10.556394 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:10.556415 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:10.556476 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:10.560491 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.561142 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.561155 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.561159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.561334 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.561715 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.561721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.561724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.562194 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:10.562488 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.562680 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.562915 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:10:10.564193 13631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:10.680450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13496
I20260619 13:10:10.698411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13640
I20260619 13:10:10.711747 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:10.699141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.97.3:35813)
0619 13:10:10.699309 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:10.699314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:10.699374 (+ 60us) server_negotiation.cc:407] Connection header received
0619 13:10:10.699543 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:10.699547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:10.699609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:10.699697 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:10.700972 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.701747 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.703020 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:10.703431 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:10.703535 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:10.706314 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:10.706329 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:10.706345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:10.706388 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:10.709332 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.710054 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.710061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.710063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.710207 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:10.710524 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:10.710527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:10.710528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:10.710874 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:10.711093 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:10.711316 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 13:10:10.711479 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:10:10.712425 13765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:10.800410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13640
I20260619 13:10:10.816601 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13262
I20260619 13:10:10.817902 13325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:10.930521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13262
I20260619 13:10:10.952147 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:10:11.297753 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:10:11.298014 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:10:11.644233 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:10:11.644 UTC [12314] LOG: received smart shutdown request
2026-06-19 13:10:11.647 UTC [12314] LOG: background worker "logical replication launcher" (PID 12323) exited with exit code 1
2026-06-19 13:10:11.647 UTC [12318] LOG: shutting down
2026-06-19 13:10:11.805 UTC [12314] LOG: database system is shut down
2026-06-19T13:10:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53841 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:10:12 dist-test-slave-q11v krb5kdc[13838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:10:12 dist-test-slave-q11v krb5kdc[13838](info): set up 2 sockets
Jun 19 13:10:12 dist-test-slave-q11v krb5kdc[13838](info): commencing operation
krb5kdc: starting...
W20260619 13:10:13.038214 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:10:13 dist-test-slave-q11v krb5kdc[13838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:10:13.273038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:13.607031 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:13.607321 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:13.607508 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:13.614916 13851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:13.615048 13851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:13.615134 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:13.615231 13851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:13.615315 13851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:13.625705 13851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:13.627094 13851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:13.628506 13851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:13.640616 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:13.640579 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:13.640672 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:13.642686 13851 server_base.cc:1034] running on GCE node
I20260619 13:10:13.643532 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:13.644945 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:13.646144 13851 hybrid_clock.cc:648] HybridClock initialized: now 1781874613646078 us; error 55 us; skew 500 ppm
I20260619 13:10:13.646471 13851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:13 dist-test-slave-q11v krb5kdc[13838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:13.649932 13851 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:13.651408 13851 webserver.cc:460] Webserver started at http://127.0.97.62:33397/ using document root <none> and password file <none>
I20260619 13:10:13.652137 13851 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:13.652282 13851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:13.652599 13851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:13.655699 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "85e79730f93e4a008d6d2f1055d17c98"
format_stamp: "Formatted at 2026-06-19 13:10:13 on dist-test-slave-q11v"
server_key: "a0680fed70e399b71812956e7d589776"
server_key_iv: "affb73b2ef5b9d4bc9dc4159c11d29e1"
server_key_version: "encryptionkey@0"
I20260619 13:10:13.656507 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "85e79730f93e4a008d6d2f1055d17c98"
format_stamp: "Formatted at 2026-06-19 13:10:13 on dist-test-slave-q11v"
server_key: "a0680fed70e399b71812956e7d589776"
server_key_iv: "affb73b2ef5b9d4bc9dc4159c11d29e1"
server_key_version: "encryptionkey@0"
I20260619 13:10:13.661778 13851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:10:13.665323 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:13.666811 13851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:13.667003 13851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "85e79730f93e4a008d6d2f1055d17c98"
format_stamp: "Formatted at 2026-06-19 13:10:13 on dist-test-slave-q11v"
server_key: "a0680fed70e399b71812956e7d589776"
server_key_iv: "affb73b2ef5b9d4bc9dc4159c11d29e1"
server_key_version: "encryptionkey@0"
I20260619 13:10:13.667223 13851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:13.692548 13851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:13.697446 13851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:13.697788 13851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:13.717161 13851 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41817
I20260619 13:10:13.717200 13917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41817 every 8 connection(s)
I20260619 13:10:13.718580 13851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:13.722544 13918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:13.726011 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13851
I20260619 13:10:13.726249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:10:13.727133 388 external_mini_cluster.cc:1428] Setting key 8a4225c75ac9b39d3238bf445772bd5c
I20260619 13:10:13.732273 13918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:10:13 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 13:10:13.758239 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.736666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41817 (local address 127.0.0.1:42518)
0619 13:10:13.737977 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.738024 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:13.739954 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:13.740851 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:13.740860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:13.741692 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:13.742063 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:13.742081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:13.743739 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:13.743745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:13.744911 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:13.744923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:13.746133 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:13.746141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:13.746373 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:13.747102 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:13.747132 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:13.749495 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:13.753966 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:13.753977 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:13.753987 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:13.754426 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:13.754936 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:13.754939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:13.754942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:13.755099 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:13.756206 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:13.756211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:13.756417 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 13:10:13.756646 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 13:10:13.757059 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":6,"spinlock_wait_cycles":57088,"thread_start_us":868,"threads_started":1}
I20260619 13:10:13.758301 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.736669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.0.97.62:41817)
0619 13:10:13.737878 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:13.737891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:13.738160 (+ 269us) server_negotiation.cc:407] Connection header received
0619 13:10:13.740365 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:13.740371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:13.740622 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:13.740842 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:13.742265 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:13.743572 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:13.745099 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:13.745865 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:13.746078 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:13.749779 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:13.749788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:13.749801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:13.749833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:13.753806 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:13.754590 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:13.754602 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:13.754618 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:13.754815 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:13.755223 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:13.755231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:13.755236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:13.756048 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:13.756252 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:13.756853 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 13:10:13.757232 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":980,"thread_start_us":564,"threads_started":1}
I20260619 13:10:13.761209 13918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:10:13.762401 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.761999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53154)
0619 13:10:13.762173 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.762279 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 13:10:13.762683 13869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
I20260619 13:10:13.764318 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.763958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53158)
0619 13:10:13.764117 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.764198 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:10:13.764537 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:10:13.779172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44863
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:13.813853 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } attempt: 1
I20260619 13:10:13.815111 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.814551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53168)
0619 13:10:13.814802 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.814942 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:10:13.816239 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.815829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53182)
0619 13:10:13.815983 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.816073 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:10:13.816450 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
I20260619 13:10:13.884770 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } attempt: 2
I20260619 13:10:13.886300 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.885547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53194)
0619 13:10:13.885938 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.886091 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 13:10:13.887581 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.887118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53206)
0619 13:10:13.887310 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.887416 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:10:13.887871 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
I20260619 13:10:13.979214 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } attempt: 3
I20260619 13:10:13.980468 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.979926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53218)
0619 13:10:13.980168 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.980286 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:10:13.981673 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:13.981200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53222)
0619 13:10:13.981362 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:13.981454 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:10:13.981901 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
W20260619 13:10:14.113219 13925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:14.113533 13925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:14.113699 13925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:14.121034 13925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:14.121145 13925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:14.121192 13925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:14.121248 13925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:14.121296 13925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:14.125162 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } attempt: 4
I20260619 13:10:14.126514 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.125874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53230)
0619 13:10:14.126197 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.126319 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 13:10:14.127624 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.127231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53232)
0619 13:10:14.127387 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.127482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:10:14.127842 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44863: connect: Connection refused (error 111)
I20260619 13:10:14.132349 13925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:14.133735 13925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:14.135027 13925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:14.152554 13933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:14.152639 13931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:14.152663 13930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:14.153672 13925 server_base.cc:1034] running on GCE node
I20260619 13:10:14.154551 13925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:14.155984 13925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:14.157186 13925 hybrid_clock.cc:648] HybridClock initialized: now 1781874614157118 us; error 68 us; skew 500 ppm
I20260619 13:10:14.157562 13925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:14.167636 13925 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:10:14.169173 13925 webserver.cc:460] Webserver started at http://127.0.97.61:32845/ using document root <none> and password file <none>
I20260619 13:10:14.169960 13925 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:14.170066 13925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:14.170382 13925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:14.174010 13925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data/instance:
uuid: "51698f5917c64ad7a373da6ec990291a"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "8a72b66976be194807b067b9d29b24fc"
server_key_iv: "606656c32455b461b7e12230961b9766"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.174777 13925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal/instance:
uuid: "51698f5917c64ad7a373da6ec990291a"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "8a72b66976be194807b067b9d29b24fc"
server_key_iv: "606656c32455b461b7e12230961b9766"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.180202 13925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:10:14.183893 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:14.185426 13925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:14.185643 13925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "51698f5917c64ad7a373da6ec990291a"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "8a72b66976be194807b067b9d29b24fc"
server_key_iv: "606656c32455b461b7e12230961b9766"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.185899 13925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:14.215330 13925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:14.219790 13925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:14.220136 13925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:14.239509 13925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44863
I20260619 13:10:14.239539 13991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44863 every 8 connection(s)
I20260619 13:10:14.240939 13925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:10:14.241976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13925
I20260619 13:10:14.242161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-1/wal/instance
I20260619 13:10:14.242614 388 external_mini_cluster.cc:1428] Setting key a0589c435c9433622d9a4d93f8b10ed6
I20260619 13:10:14.245100 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:14.251945 13992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:14.275455 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.254857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.0.97.62:41817)
0619 13:10:14.255040 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.255045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.256546 (+ 1501us) server_negotiation.cc:407] Connection header received
0619 13:10:14.258541 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.258548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.258645 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.258768 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.260318 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.261004 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.262817 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.263356 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.263527 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.268149 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.268156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.268159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.268195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.271543 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.272710 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.272720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.272726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.272887 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.273319 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.273327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.273333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.273811 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.273967 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.274743 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.275030 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:10:14.276182 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.255519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41817 (local address 127.0.0.1:42530)
0619 13:10:14.256370 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.256408 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.258232 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.259013 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.259023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.259562 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.260138 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.260156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.261172 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.261176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.262665 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.262678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.263686 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.263697 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.263969 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.264711 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.264747 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.267870 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.271816 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.271835 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.271851 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.272548 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.273007 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.273011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.273014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.273201 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.273944 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.273951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.274325 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.274563 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.274979 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":370,"threads_started":1}
I20260619 13:10:14.278041 13992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:10:14.293244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35553
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:14.295287 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.278883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53242)
0619 13:10:14.279079 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.279100 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.279226 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.279972 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.279976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.280444 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.280573 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.280582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.282128 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.282139 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.283423 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.283435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.284284 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.284290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.284541 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.285162 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.285180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.288677 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.292588 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.292593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.292596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.293021 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.293464 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.293471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.293477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.293693 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.294516 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.294520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.294667 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.294794 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.295008 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260619 13:10:14.295765 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.278963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.0.97.61:44863)
0619 13:10:14.279549 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.279555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.279579 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:10:14.279621 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.279626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.279826 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.279986 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.280727 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.281985 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.283557 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.284116 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.284270 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.288961 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.288980 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.289005 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.289041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.292435 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.293166 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.293176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.293181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.293331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.293805 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.293819 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.293825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.294362 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.294578 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.294991 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.295285 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
I20260619 13:10:14.298360 13992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:10:14.299635 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.299166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37468)
0619 13:10:14.299357 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.299490 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 13:10:14.299993 13942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.301817 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.301390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37480)
0619 13:10:14.301557 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.301703 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 13:10:14.302052 13992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.351362 13992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 1
I20260619 13:10:14.352595 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.352127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37494)
0619 13:10:14.352334 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.352444 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 13:10:14.353714 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.353253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37506)
0619 13:10:14.353425 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.353493 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 13:10:14.353936 13992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.422149 13992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 2
I20260619 13:10:14.423362 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.422857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37520)
0619 13:10:14.423082 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.423190 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 13:10:14.424389 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.424065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37526)
0619 13:10:14.424208 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.424275 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:10:14.424576 13992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.427069 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } attempt: 5
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:10:14.440604 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.427789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53252)
0619 13:10:14.427985 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.428007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.428180 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.428548 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.428554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.428978 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.429159 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.429169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.430251 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.430256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.431118 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.431129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.431919 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.431926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.432115 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.432778 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.432807 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.434862 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.438181 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.438187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.438191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.438575 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.438940 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.438945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.438948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.439086 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.439615 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.439620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.439778 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.439939 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.440234 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 13:10:14.440599 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.427821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.0.97.61:44863)
0619 13:10:14.428001 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.428010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.428096 (+ 86us) server_negotiation.cc:407] Connection header received
0619 13:10:14.428344 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.428348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.428409 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.428512 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.429315 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.430085 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.431261 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.431717 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.431884 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.435051 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.435056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.435059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.435091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.438039 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.438698 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.438706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.438709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.438829 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.439195 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.439201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.439204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.439498 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.439654 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.440045 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.440261 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:10:14.441972 13918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:10:14.443050 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.442664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37536)
0619 13:10:14.442834 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.442923 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:10:14.443997 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.443683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37542)
0619 13:10:14.443821 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.443904 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:10:14.444311 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.495545 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 1
I20260619 13:10:14.496857 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.496371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37548)
0619 13:10:14.496579 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.496684 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:10:14.498061 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.497731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37564)
0619 13:10:14.497882 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.497951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 13:10:14.498325 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.515784 13992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 3
I20260619 13:10:14.517164 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.516463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37568)
0619 13:10:14.516783 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.516934 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 13:10:14.518492 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.517962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37572)
0619 13:10:14.518218 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.518328 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260619 13:10:14.518708 13992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.566558 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 2
I20260619 13:10:14.567814 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.567336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37588)
0619 13:10:14.567561 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.567651 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:10:14.568769 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.568449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37602)
0619 13:10:14.568594 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.568670 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:10:14.568981 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
W20260619 13:10:14.641424 13999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:14.641827 13999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:14.641980 13999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:14.649295 13999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:14.649437 13999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:14.649537 13999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:14.649652 13999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:14.649744 13999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:14.659689 13999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41817,127.0.97.61:44863,127.0.97.60:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:14.661118 13999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:14.661975 13992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 4
I20260619 13:10:14.662599 13999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:10:14.663090 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.662622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37618)
0619 13:10:14.662864 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.662947 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 13:10:14.664096 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.663761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37634)
0619 13:10:14.663925 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.663986 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:10:14.664291 13992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.675164 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 3
W20260619 13:10:14.675243 14005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:14.675273 14007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:14.675999 13999 server_base.cc:1034] running on GCE node
W20260619 13:10:14.676061 14004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:14.676362 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.675751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37648)
0619 13:10:14.676026 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.676142 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:10:14.677270 13999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:14.677467 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.677010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37650)
0619 13:10:14.677229 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.677331 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 13:10:14.677754 13918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35553: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35553: connect: Connection refused (error 111)
I20260619 13:10:14.678896 13999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:14.680109 13999 hybrid_clock.cc:648] HybridClock initialized: now 1781874614680056 us; error 50 us; skew 500 ppm
I20260619 13:10:14.680488 13999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:14.683864 13999 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:10:14.685437 13999 webserver.cc:460] Webserver started at http://127.0.97.60:42823/ using document root <none> and password file <none>
I20260619 13:10:14.686189 13999 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:14.686322 13999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:14.686635 13999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:14.689877 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data/instance:
uuid: "3c35383b79da4576b57150866e7b8fd7"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "d4211806c4bfd3d44d22c1a5c8ddba4a"
server_key_iv: "7413246fb3b54ebdb363cebd6f1170ac"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.690829 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal/instance:
uuid: "3c35383b79da4576b57150866e7b8fd7"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "d4211806c4bfd3d44d22c1a5c8ddba4a"
server_key_iv: "7413246fb3b54ebdb363cebd6f1170ac"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.696086 13999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:10:14.699560 14014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:14.700918 13999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:14.701129 13999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "3c35383b79da4576b57150866e7b8fd7"
format_stamp: "Formatted at 2026-06-19 13:10:14 on dist-test-slave-q11v"
server_key: "d4211806c4bfd3d44d22c1a5c8ddba4a"
server_key_iv: "7413246fb3b54ebdb363cebd6f1170ac"
server_key_version: "encryptionkey@0"
I20260619 13:10:14.701354 13999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:14.718767 13999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:14.723265 13999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:14.723562 13999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:14.742372 13999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35553
I20260619 13:10:14.742393 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35553 every 8 connection(s)
I20260619 13:10:14.743696 13999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:10:14.747365 14066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:14.748040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 13999
I20260619 13:10:14.748217 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874405339864-388-0/minicluster-data/master-2/wal/instance
I20260619 13:10:14.748649 388 external_mini_cluster.cc:1428] Setting key fe0b322cee95f9fe6708eb8fe2f79060
I20260619 13:10:14.757190 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:14.774612 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.750562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.97.60:35553)
0619 13:10:14.751384 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.751399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.751456 (+ 57us) server_negotiation.cc:407] Connection header received
0619 13:10:14.754026 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.754055 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.754593 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.755056 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.756424 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.758434 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.760077 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.761143 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.761338 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.766232 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.766261 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.766278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.766326 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.769998 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.770904 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.770915 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.770935 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.771134 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.771591 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.771596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.771599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.772264 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.772576 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.773036 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.773498 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":322,"threads_started":1}
I20260619 13:10:14.776060 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.760105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.97.62:41817)
0619 13:10:14.760335 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.760343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.761172 (+ 829us) server_negotiation.cc:407] Connection header received
0619 13:10:14.761349 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.761355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.761441 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.761550 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.762655 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.763820 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.765459 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.766195 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.766285 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.770234 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.770240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.770243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.770276 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.773060 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.773948 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.773954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.773958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.774101 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.774484 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.774488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.774490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.774811 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.774986 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.775455 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.775672 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:10:14.776115 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.760134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41817 (local address 127.0.0.1:42538)
0619 13:10:14.761029 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.761073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.761232 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.761646 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.761653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.762213 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.762500 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.762509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.763976 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.763980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.765305 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.765317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.766371 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.766381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.766590 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.767282 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.767308 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.770038 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.773197 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.773210 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.773217 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.773832 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.774216 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.774221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.774225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.774363 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.774961 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.774969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.775161 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.775335 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.775648 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":449,"threads_started":1}
I20260619 13:10:14.777549 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:14.790659 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.778345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44863 (local address 127.0.0.1:53260)
0619 13:10:14.778509 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.778525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.778627 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.778929 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.778933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.779296 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.779435 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.779443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.780403 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.780409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.781272 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.781282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.782085 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.782091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.782278 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.782901 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.782927 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.784912 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.788198 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.788205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.788209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.788592 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.788971 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.788976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.788979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.789124 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.789694 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.789698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.789902 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.790064 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.790342 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:10:14.790830 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.778318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.0.97.61:44863)
0619 13:10:14.778474 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.778479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.778588 (+ 109us) server_negotiation.cc:407] Connection header received
0619 13:10:14.778728 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.778733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.778807 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.778909 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.779578 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.780261 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.781428 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.781935 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.782072 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.785099 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.785104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.785107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.785139 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.788070 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.788717 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.788727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.788729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.788860 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.789232 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.789238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.789241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.789515 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.789752 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.790202 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.790438 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 13:10:14.792037 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:10:14.792308 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.777841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.0.97.62:41817)
0619 13:10:14.778023 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.778031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.778276 (+ 245us) server_negotiation.cc:407] Connection header received
0619 13:10:14.778486 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.778491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.778567 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.778672 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.779838 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.780559 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.781905 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.782293 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.782429 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.786428 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.786433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.786436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.786474 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.788944 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.789592 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.789633 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.789637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.789793 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.790217 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.790221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.790224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.790579 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.790875 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.791760 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.792006 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:10:14.805994 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.792826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.97.60:35553)
0619 13:10:14.793023 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.793028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.793234 (+ 206us) server_negotiation.cc:407] Connection header received
0619 13:10:14.793388 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.793392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.793458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.793555 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.794313 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.795140 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.796518 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.797021 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.797110 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.800398 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.800404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.800408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.800440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.803433 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.804026 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.804033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.804037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.804145 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.804492 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.804499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.804504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.804775 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.804931 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.805329 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.805564 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":65}
I20260619 13:10:14.806106 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.792835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37674)
0619 13:10:14.793124 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.793142 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.793258 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.793567 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.793571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.793996 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.794167 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.794182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.795275 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.795282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.796386 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.796398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.797178 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.797184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.797389 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.797972 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.797994 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.800231 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.803560 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.803568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.803572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.803912 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.804243 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.804248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.804252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.804393 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.804894 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.804898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.805078 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.805248 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.805734 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":147}
I20260619 13:10:14.809325 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.793981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.0.97.61:44863)
0619 13:10:14.794129 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.794134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.794206 (+ 72us) server_negotiation.cc:407] Connection header received
0619 13:10:14.794389 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.794395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.794464 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.794562 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.795835 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.796590 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.798155 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.798561 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.798653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.803376 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.803381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.803384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.803416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.806444 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.807276 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.807282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.807285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.807456 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.807883 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.807889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.807892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.808177 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.808458 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.808775 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.809027 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:10:14.811975 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: Bootstrap starting.
I20260619 13:10:14.814165 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:10:14.815621 14066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:14.816797 14066 log.cc:824] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:14.820294 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: No bootstrap required, opened a new log
I20260619 13:10:14.836144 14066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:14.837517 14066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:14.837795 14066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c35383b79da4576b57150866e7b8fd7, State: Initialized, Role: FOLLOWER
I20260619 13:10:14.838901 14066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:14.846199 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:14.848717 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:14.848572 14066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:14.853101 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.819536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.97.60:35553)
0619 13:10:14.823747 (+ 4211us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.823757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.830954 (+ 7197us) server_negotiation.cc:407] Connection header received
0619 13:10:14.831615 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.831624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.831735 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.831840 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.835116 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.836607 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.840438 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.841216 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.841356 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.845376 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.845383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.845388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.845430 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.849446 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.850442 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.850447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.850451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.850603 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.851133 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.851138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.851141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.851549 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.851813 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.852448 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.852660 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3999}
I20260619 13:10:14.856568 13918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 4
I20260619 13:10:14.857520 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.819121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.97.62:41817)
0619 13:10:14.819512 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.819518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.839891 (+ 20373us) server_negotiation.cc:407] Connection header received
0619 13:10:14.840171 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.840178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.840268 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.840340 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.842540 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.843695 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.845787 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.846513 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.846612 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.850042 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.850051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.850056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.850098 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.854001 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.854825 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.854832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.854837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.855003 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.855458 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.855464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.855469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.855810 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.856071 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.856691 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.857004 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260619 13:10:14.861778 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.819300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.0.97.61:44863)
0619 13:10:14.825525 (+ 6225us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.825534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.840266 (+ 14732us) server_negotiation.cc:407] Connection header received
0619 13:10:14.840402 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.840410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.840517 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.840645 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.843579 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.844961 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.847217 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.847849 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.847971 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.851981 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.851990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.851997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.852053 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.856404 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.859181 (+ 2777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.859187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.859191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.859356 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.859823 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.859832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.859838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.860298 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.860590 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.860992 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.861316 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6021}
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 13:10:14.874049 14090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:10:14.874266 14090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:10:14.875600 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.857147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37688)
0619 13:10:14.857393 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.857411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.857549 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.857945 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.857949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.858489 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.858708 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.858721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.860259 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.860265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.861733 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.861751 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.862640 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.862649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.862916 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.863602 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.863637 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.866856 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.872447 (+ 5591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.872453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.872457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.872991 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.873491 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.873499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.873504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.873717 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.874493 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.874501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.874675 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.874835 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.875184 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 13:10:14.876479 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.857183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.0.97.60:35553)
0619 13:10:14.857326 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.857332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.857491 (+ 159us) server_negotiation.cc:407] Connection header received
0619 13:10:14.857742 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.857747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.857832 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.857920 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.858891 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.860077 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.861892 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.862459 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.862597 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.867094 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.867101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.867106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.867148 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.872266 (+ 5118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.873178 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.873183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.873186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.873345 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.873896 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.873901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.873905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.874329 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.874527 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.874981 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.876090 (+ 1109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:10:14.882344 13918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: Bootstrap starting.
I20260619 13:10:14.870833 13999 master_runner.cc:186] Error getting master registration for 127.0.97.62:41817: OK, instance_id { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" instance_seqno: 1781874613711318 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:10:14.882829 13999 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:10:14.886888 13918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:14.887948 13918 log.cc:824] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:14.890925 13918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: No bootstrap required, opened a new log
I20260619 13:10:14.898999 13918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:14.899494 13918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:14.899682 13918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e79730f93e4a008d6d2f1055d17c98, State: Initialized, Role: FOLLOWER
I20260619 13:10:14.900565 13918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:14.901513 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:14.901916 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:14.902724 13918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:10:14.914608 14104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:10:14.914832 14104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:10:14.913466 13851 master_runner.cc:186] Error getting master registration for 127.0.97.61:44863: OK, instance_id { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" instance_seqno: 1781874614233907 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:10:14.926424 13851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 13:10:14.950768 13872 tablet.cc:2369] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:14.963515 13992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } attempt: 5
Jun 19 13:10:14 dist-test-slave-q11v krb5kdc[13838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 13:10:14.975615 14020 tablet.cc:2369] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:14.982882 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.964209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35553 (local address 127.0.0.1:37694)
0619 13:10:14.964463 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:14.964484 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:14.964589 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:14.964972 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:14.964977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:14.965507 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:14.965747 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.965761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.967198 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.967205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.968673 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:14.968688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.969755 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.969762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:14.969978 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.970712 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:14.970741 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:14.973825 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:14.979966 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.979973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.979978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.980536 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.980964 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:14.980972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:14.980976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:14.981149 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:14.981828 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:14.981833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:14.982010 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 13:10:14.982174 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 13:10:14.982488 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 13:10:14.982972 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:14.964302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.0.97.60:35553)
0619 13:10:14.964510 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:14.964516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:14.964537 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:10:14.964723 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:14.964730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:14.964813 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:14.964920 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:14.965902 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.967035 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.968823 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:14.969547 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:14.969727 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:14.974047 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:14.974054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:14.974059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:14.974106 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:14.979807 (+ 5701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.980674 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.980680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.980684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.980848 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:14.981262 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:14.981268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:14.981272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:14.981700 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:14.981861 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:14.982282 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 13:10:14.982552 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 13:10:14.989234 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: Bootstrap starting.
I20260619 13:10:14.993628 13992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:14.995036 13992 log.cc:824] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:14.999455 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: No bootstrap required, opened a new log
I20260619 13:10:15.010998 13992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:15.011555 13992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:15.011744 13992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51698f5917c64ad7a373da6ec990291a, State: Initialized, Role: FOLLOWER
I20260619 13:10:15.012522 13992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:15.014453 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.014955 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:15.015937 13992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:15.036331 13925 master_runner.cc:263] No leader master found from master 51698f5917c64ad7a373da6ec990291a
I20260619 13:10:15.036510 13925 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 13:10:15.044149 14118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:10:15.044410 14118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:10:15.049860 14093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:10:15.050194 14093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:15.051851 14093 leader_election.cc:288] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51698f5917c64ad7a373da6ec990291a (127.0.97.61:44863), 3c35383b79da4576b57150866e7b8fd7 (127.0.97.60:35553)
I20260619 13:10:15.053021 13967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e79730f93e4a008d6d2f1055d17c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51698f5917c64ad7a373da6ec990291a" is_pre_election: true
I20260619 13:10:15.056651 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e79730f93e4a008d6d2f1055d17c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c35383b79da4576b57150866e7b8fd7" is_pre_election: true
I20260619 13:10:15.056995 13967 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e79730f93e4a008d6d2f1055d17c98 in term 0.
I20260619 13:10:15.057277 14041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e79730f93e4a008d6d2f1055d17c98 in term 0.
I20260619 13:10:15.057983 13868 leader_election.cc:302] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c35383b79da4576b57150866e7b8fd7, 51698f5917c64ad7a373da6ec990291a, 85e79730f93e4a008d6d2f1055d17c98; no voters:
I20260619 13:10:15.058737 14093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:10:15.058892 14093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:10:15.058985 14093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:15.062546 14093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:15.063758 13967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e79730f93e4a008d6d2f1055d17c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51698f5917c64ad7a373da6ec990291a"
I20260619 13:10:15.063985 13967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:15.067513 13967 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e79730f93e4a008d6d2f1055d17c98 in term 1.
I20260619 13:10:15.068048 13869 leader_election.cc:302] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51698f5917c64ad7a373da6ec990291a, 85e79730f93e4a008d6d2f1055d17c98; no voters:
I20260619 13:10:15.072240 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e79730f93e4a008d6d2f1055d17c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c35383b79da4576b57150866e7b8fd7"
I20260619 13:10:15.072489 14041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:15.076033 14041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e79730f93e4a008d6d2f1055d17c98 in term 1.
I20260619 13:10:15.080921 14093 leader_election.cc:288] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [CANDIDATE]: Term 1 election: Requested vote from peers 51698f5917c64ad7a373da6ec990291a (127.0.97.61:44863), 3c35383b79da4576b57150866e7b8fd7 (127.0.97.60:35553)
I20260619 13:10:15.081409 14093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:15.081866 14093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [term 1 LEADER]: Becoming Leader. State: Replica: 85e79730f93e4a008d6d2f1055d17c98, State: Running, Role: LEADER
I20260619 13:10:15.082737 14093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } }
I20260619 13:10:15.096057 14120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e79730f93e4a008d6d2f1055d17c98. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.096652 14120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:15.108285 14122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:15.110888 14122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:15.132313 14041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [term 1 FOLLOWER]: Refusing update from remote peer 85e79730f93e4a008d6d2f1055d17c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:15.133384 14120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:15.140794 13967 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [term 1 FOLLOWER]: Refusing update from remote peer 85e79730f93e4a008d6d2f1055d17c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:15.141863 14120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:15.177219 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e79730f93e4a008d6d2f1055d17c98. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.177573 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:15.181849 14123 mvcc.cc:205] Tried to move back new op lower bound from 7298558423549841408 to 7298558423391739904. Current Snapshot: MvccSnapshot[applied={T|T < 7298558423549841408}]
I20260619 13:10:15.183229 14124 mvcc.cc:205] Tried to move back new op lower bound from 7298558423549841408 to 7298558423391739904. Current Snapshot: MvccSnapshot[applied={T|T < 7298558423549841408}]
I20260619 13:10:15.188795 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e79730f93e4a008d6d2f1055d17c98. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.189144 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:15.195286 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.195612 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:15.195576 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.195835 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:15.195899 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.196148 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:15.196873 14122 catalog_manager.cc:1347] Generated new cluster ID: 67ab49e240074fe997f16ae7405a7bbb
I20260619 13:10:15.196972 14122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:15.196975 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e79730f93e4a008d6d2f1055d17c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e79730f93e4a008d6d2f1055d17c98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41817 } } peers { permanent_uuid: "51698f5917c64ad7a373da6ec990291a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44863 } } peers { permanent_uuid: "3c35383b79da4576b57150866e7b8fd7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35553 } } }
I20260619 13:10:15.197227 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:10:15.224910 14122 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:10:15.226902 14122 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:15.257455 14122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e79730f93e4a008d6d2f1055d17c98: Generated new TSK 0
I20260619 13:10:15.258728 14122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:10:15.876152 14090 catalog_manager.cc:1259] Loaded cluster ID: 67ab49e240074fe997f16ae7405a7bbb
I20260619 13:10:15.876292 14090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: loading cluster ID for follower catalog manager: success
I20260619 13:10:15.878818 14090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: acquiring CA information for follower catalog manager: success
I20260619 13:10:15.880587 14090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c35383b79da4576b57150866e7b8fd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:10:16.046607 14118 catalog_manager.cc:1259] Loaded cluster ID: 67ab49e240074fe997f16ae7405a7bbb
I20260619 13:10:16.046712 14118 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: loading cluster ID for follower catalog manager: success
I20260619 13:10:16.049278 14118 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: acquiring CA information for follower catalog manager: success
I20260619 13:10:16.051330 14118 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51698f5917c64ad7a373da6ec990291a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:10:16.418696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13851
I20260619 13:10:16.419935 13913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:16.531980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13851
W20260619 13:10:16.553102 14018 connection.cc:535] server connection from 127.0.0.1:37688 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:10:16.553754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13925
I20260619 13:10:16.554872 13987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:16.663565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13925
I20260619 13:10:16.683869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 13999
I20260619 13:10:16.684978 14061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:16.793924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 13999
2026-06-19T13:10:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4863 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:10:16 dist-test-slave-q11v krb5kdc[14141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:10:16 dist-test-slave-q11v krb5kdc[14141](info): set up 2 sockets
Jun 19 13:10:16 dist-test-slave-q11v krb5kdc[14141](info): commencing operation
krb5kdc: starting...
W20260619 13:10:17.914214 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:10:18.144896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42787
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:18.478263 14154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:18.478650 14154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:18.478834 14154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:18.486116 14154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:18.486274 14154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:18.486387 14154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:18.486501 14154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:18.486588 14154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:18.497205 14154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:18.498672 14154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:18.500056 14154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:18.515468 14159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:18.515865 14162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:18.516404 14160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:18.517342 14154 server_base.cc:1034] running on GCE node
I20260619 13:10:18.518545 14154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:18.520414 14154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:18.521662 14154 hybrid_clock.cc:648] HybridClock initialized: now 1781874618521605 us; error 36 us; skew 500 ppm
I20260619 13:10:18.522110 14154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:18.525966 14154 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:18.528105 14154 webserver.cc:460] Webserver started at http://127.0.97.62:39493/ using document root <none> and password file <none>
I20260619 13:10:18.529060 14154 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:18.529206 14154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:18.529654 14154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:18.534114 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "3c556440346346d89cec7411cbfca407"
format_stamp: "Formatted at 2026-06-19 13:10:18 on dist-test-slave-q11v"
server_key: "0f0c99e4828b3f11755c6180c038f707"
server_key_iv: "59b6c314deda4ec6117c6f22cd144e72"
server_key_version: "encryptionkey@0"
I20260619 13:10:18.535012 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c556440346346d89cec7411cbfca407"
format_stamp: "Formatted at 2026-06-19 13:10:18 on dist-test-slave-q11v"
server_key: "0f0c99e4828b3f11755c6180c038f707"
server_key_iv: "59b6c314deda4ec6117c6f22cd144e72"
server_key_version: "encryptionkey@0"
I20260619 13:10:18.542268 14154 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 13:10:18.547145 14169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:18.548856 14154 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:10:18.549063 14154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "3c556440346346d89cec7411cbfca407"
format_stamp: "Formatted at 2026-06-19 13:10:18 on dist-test-slave-q11v"
server_key: "0f0c99e4828b3f11755c6180c038f707"
server_key_iv: "59b6c314deda4ec6117c6f22cd144e72"
server_key_version: "encryptionkey@0"
I20260619 13:10:18.549329 14154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:18.582777 14154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:18.587360 14154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:18.587682 14154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:18.606802 14154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42787
I20260619 13:10:18.606827 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42787 every 8 connection(s)
I20260619 13:10:18.608295 14154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:18.609565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14154
I20260619 13:10:18.609810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:10:18.610678 388 external_mini_cluster.cc:1428] Setting key 2526b3cea8a1153b5f764baaea12dd2d
I20260619 13:10:18.613380 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:18.627626 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407: Bootstrap starting.
I20260619 13:10:18.631114 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:18.632244 14221 log.cc:824] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:18.635702 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407: No bootstrap required, opened a new log
I20260619 13:10:18.635936 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:18.613072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.0.97.62:42787)
0619 13:10:18.613960 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:18.613973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:18.614041 (+ 68us) server_negotiation.cc:407] Connection header received
0619 13:10:18.615764 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:18.615787 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:18.616215 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:18.616579 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:18.618600 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:18.619968 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:18.621911 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:18.622606 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:18.622738 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:18.627803 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:18.627814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:18.627837 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:18.627892 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:18.631124 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:18.632000 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:18.632008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:18.632014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:18.632204 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:18.632665 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:18.632670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:18.632672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:18.633576 (+ 904us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:18.633920 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:18.634424 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 13:10:18.634847 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":467,"threads_started":1}
I20260619 13:10:18.640285 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:10:18.644601 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } }
I20260619 13:10:18.645054 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:18.645174 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c556440346346d89cec7411cbfca407, State: Initialized, Role: FOLLOWER
I20260619 13:10:18.645875 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } }
I20260619 13:10:18.646144 14221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:18.646292 14221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:18.646483 14221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:18.648756 14221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } }
I20260619 13:10:18.649303 14221 leader_election.cc:302] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c556440346346d89cec7411cbfca407; no voters:
I20260619 13:10:18.649999 14221 leader_election.cc:288] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:10:18.650173 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:18.651067 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [term 1 LEADER]: Becoming Leader. State: Replica: 3c556440346346d89cec7411cbfca407, State: Running, Role: LEADER
I20260619 13:10:18.651608 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } }
I20260619 13:10:18.652283 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:18.654678 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c556440346346d89cec7411cbfca407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } } }
I20260619 13:10:18.655248 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:18.655148 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c556440346346d89cec7411cbfca407. Latest consensus state: current_term: 1 leader_uuid: "3c556440346346d89cec7411cbfca407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c556440346346d89cec7411cbfca407" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42787 } } }
I20260619 13:10:18.655449 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:18.657493 14235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:18.662413 14235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:18.671823 14235 catalog_manager.cc:1347] Generated new cluster ID: c99ee9d4eb80494685ccfc979d045ee7
I20260619 13:10:18.671963 14235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:18.693156 14235 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:10:18.695261 14235 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:18.701958 14235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c556440346346d89cec7411cbfca407: Generated new TSK 0
I20260619 13:10:18.703055 14235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:10:18.752431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14154
I20260619 13:10:18.753839 14216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:18.855799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14154
2026-06-19T13:10:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2060 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14249](info): set up 2 sockets
Jun 19 13:10:18 dist-test-slave-q11v krb5kdc[14249](info): commencing operation
krb5kdc: starting...
W20260619 13:10:19.954051 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:10:20 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:10:20.190750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:20.554193 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:20.554632 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:20.554795 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:20.562407 14262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:20.562788 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:20.562927 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:20.563055 14262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:20.563143 14262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:20.574788 14262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:20.576356 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:20.577935 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:20.591249 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:20.591202 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:20.591617 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:20.592059 14262 server_base.cc:1034] running on GCE node
I20260619 13:10:20.593003 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:20.594688 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:20.595947 14262 hybrid_clock.cc:648] HybridClock initialized: now 1781874620595910 us; error 65 us; skew 500 ppm
I20260619 13:10:20.596351 14262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:20 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:20.604242 14262 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:20.606038 14262 webserver.cc:460] Webserver started at http://127.0.97.62:38457/ using document root <none> and password file <none>
I20260619 13:10:20.606794 14262 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:20.606946 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:20.607265 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:20.610476 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "f0856d15dabe4c93b33dde41f46708b6"
format_stamp: "Formatted at 2026-06-19 13:10:20 on dist-test-slave-q11v"
server_key: "cbe6be9b5899d435ff36364aedf3f82b"
server_key_iv: "f257223e21252f10cb0dd3fd28ede3b4"
server_key_version: "encryptionkey@0"
I20260619 13:10:20.611263 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0856d15dabe4c93b33dde41f46708b6"
format_stamp: "Formatted at 2026-06-19 13:10:20 on dist-test-slave-q11v"
server_key: "cbe6be9b5899d435ff36364aedf3f82b"
server_key_iv: "f257223e21252f10cb0dd3fd28ede3b4"
server_key_version: "encryptionkey@0"
I20260619 13:10:20.616632 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:10:20.621383 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:20.622848 14262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:20.623060 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f0856d15dabe4c93b33dde41f46708b6"
format_stamp: "Formatted at 2026-06-19 13:10:20 on dist-test-slave-q11v"
server_key: "cbe6be9b5899d435ff36364aedf3f82b"
server_key_iv: "f257223e21252f10cb0dd3fd28ede3b4"
server_key_version: "encryptionkey@0"
I20260619 13:10:20.623276 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:20.642490 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:20.647100 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:20.647411 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:20.665642 14262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40361
I20260619 13:10:20.665680 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40361 every 8 connection(s)
I20260619 13:10:20.667450 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:20.671464 14329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:20.677587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14262
I20260619 13:10:20.677884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:10:20.678576 388 external_mini_cluster.cc:1428] Setting key e1cc94b172b3fe1fd51c1c60c7d9d201
I20260619 13:10:20.683066 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap starting.
I20260619 13:10:20.686311 14329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:20.687381 14329 log.cc:824] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:20.690332 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: No bootstrap required, opened a new log
Jun 19 13:10:20 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:20.698756 14329 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:20.699180 14329 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:20.699314 14329 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Initialized, Role: FOLLOWER
I20260619 13:10:20.699908 14329 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:20.700161 14329 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:20.700325 14329 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:20.700526 14329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:20.702596 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:20.680692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.97.62:40361)
0619 13:10:20.681526 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:20.681539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:20.681583 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:10:20.683403 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:20.683426 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:20.683825 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:20.684225 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:20.685696 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:20.687108 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:20.688905 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:20.689583 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:20.689849 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:20.694367 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:20.694375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:20.694393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:20.694434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:20.697755 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:20.698486 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:20.698498 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:20.698504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:20.698674 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:20.699063 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:20.699068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:20.699071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:20.699828 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:20.700142 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:20.700634 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 13:10:20.701086 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":389,"threads_started":1}
I20260619 13:10:20.703917 14329 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:20.704494 14329 leader_election.cc:302] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0856d15dabe4c93b33dde41f46708b6; no voters:
I20260619 13:10:20.705186 14329 leader_election.cc:288] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:10:20.705346 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:20.706732 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 LEADER]: Becoming Leader. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Running, Role: LEADER
I20260619 13:10:20.707237 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:20.707558 14329 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:20.710233 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0856d15dabe4c93b33dde41f46708b6. Latest consensus state: current_term: 1 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:20.710207 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:20.710760 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:20.710760 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:20.713285 14340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:20.718912 14340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:20.730288 14340 catalog_manager.cc:1347] Generated new cluster ID: 4a1de8d14ee64af4bf4f00b3f2529592
I20260619 13:10:20.730444 14340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:20.741978 14340 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:10:20.743333 14340 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:20.758555 14340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Generated new TSK 0
I20260619 13:10:20.759687 14340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:10:20.823134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:21.155447 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:21.155728 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:21.155823 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:21.163009 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:21.163126 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:21.163239 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:21.173094 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:21.174567 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:21.176204 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:21.189692 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:21.189617 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:21.189570 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:21.190328 14355 server_base.cc:1034] running on GCE node
I20260619 13:10:21.190987 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:21.191885 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:21.193090 14355 hybrid_clock.cc:648] HybridClock initialized: now 1781874621193039 us; error 43 us; skew 500 ppm
I20260619 13:10:21.193375 14355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:21 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:21.196585 14355 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:21.198249 14355 webserver.cc:460] Webserver started at http://127.0.97.1:38589/ using document root <none> and password file <none>
I20260619 13:10:21.198964 14355 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:21.199115 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:21.199425 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:21.202414 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "51ae5600cac2420ca7d5a77bc11abf72"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "c2efed587600c9244088af4d9d40933e"
server_key_iv: "1ea78841296dc5d10f7cb18c154e093b"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.203212 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "51ae5600cac2420ca7d5a77bc11abf72"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "c2efed587600c9244088af4d9d40933e"
server_key_iv: "1ea78841296dc5d10f7cb18c154e093b"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.208755 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 13:10:21.212270 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.213661 14355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:21.213835 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "51ae5600cac2420ca7d5a77bc11abf72"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "c2efed587600c9244088af4d9d40933e"
server_key_iv: "1ea78841296dc5d10f7cb18c154e093b"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.214051 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:21.230836 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:21.235210 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:21.235527 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:21.236557 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:21.238003 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:21.238111 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.238260 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:21.238335 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.271973 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37879
I20260619 13:10:21.272029 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37879 every 8 connection(s)
I20260619 13:10:21.273331 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:21.276015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14355
I20260619 13:10:21.276235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:10:21.277040 388 external_mini_cluster.cc:1428] Setting key e8c5c7725c2ae30e6aa28567b76ab914
Jun 19 13:10:21 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:21.294775 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:21.276105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39865 (local address 127.0.97.62:40361)
0619 13:10:21.276626 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:21.276632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:21.277785 (+ 1153us) server_negotiation.cc:407] Connection header received
0619 13:10:21.279675 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:21.279681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:21.279766 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:21.279897 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:21.281418 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.282632 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.284218 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.284973 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.285134 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:21.288788 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:21.288792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:21.288796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:21.288830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:21.291500 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:21.292280 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:21.292292 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:21.292295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:21.292411 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:21.292854 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:21.292861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:21.292864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:21.293149 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:21.293310 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:21.294160 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 13:10:21.294393 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260619 13:10:21.295800 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:21.276708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:39865)
0619 13:10:21.277563 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:21.277644 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:21.279395 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:21.280116 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:21.280124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:21.280643 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:21.281239 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:21.281257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.282783 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.282787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:21.284059 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:21.284071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.285196 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.285201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:21.285410 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:21.286211 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:21.286234 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:21.288551 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:21.291654 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:21.291664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:21.291682 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:21.292149 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:21.292529 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:21.292534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:21.292537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:21.292739 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:21.293284 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:21.293293 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:21.293718 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 13:10:21.294008 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 13:10:21.294582 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":370,"threads_started":1}
I20260619 13:10:21.297438 14483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:21.297780 14483 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:21.298533 14483 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:10:21.300649 14294 ts_manager.cc:194] Registered new tserver with Master: 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:10:21.302217 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:10:21.328917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:21.672868 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:21.673245 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:21.673395 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:21.685139 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:21.685288 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:21.685470 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:21.696907 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:21.698300 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:21.699615 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:21.713318 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:21.713268 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:21.713363 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:21.714177 14489 server_base.cc:1034] running on GCE node
I20260619 13:10:21.714819 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:21.715989 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:21.717253 14489 hybrid_clock.cc:648] HybridClock initialized: now 1781874621717183 us; error 55 us; skew 500 ppm
I20260619 13:10:21.717697 14489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:21 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:21.721031 14489 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:21.722609 14489 webserver.cc:460] Webserver started at http://127.0.97.2:34087/ using document root <none> and password file <none>
I20260619 13:10:21.723337 14489 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:21.723495 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:21.723826 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:21.727051 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "de7031b9c04749bd823c462adfeb0e61"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "12e153ba753ab5160a08413f14f2d7c0"
server_key_iv: "ff13d972a95643b82912bd74897ffd71"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.727955 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "de7031b9c04749bd823c462adfeb0e61"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "12e153ba753ab5160a08413f14f2d7c0"
server_key_iv: "ff13d972a95643b82912bd74897ffd71"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.733409 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:10:21.737183 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.738716 14489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:21.738921 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "de7031b9c04749bd823c462adfeb0e61"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "12e153ba753ab5160a08413f14f2d7c0"
server_key_iv: "ff13d972a95643b82912bd74897ffd71"
server_key_version: "encryptionkey@0"
I20260619 13:10:21.739142 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:21.755939 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:21.760352 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:21.760665 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:21.761792 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:21.763150 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:21.763252 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.763406 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:21.763504 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:21.797271 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32771
I20260619 13:10:21.797319 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32771 every 8 connection(s)
I20260619 13:10:21.798650 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:21.801980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14489
I20260619 13:10:21.802194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:10:21.802645 388 external_mini_cluster.cc:1428] Setting key 38cb79905f109f3c20226b153ed8fdea
Jun 19 13:10:21 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:21.822204 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:21.801346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44231 (local address 127.0.97.62:40361)
0619 13:10:21.801940 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:21.801948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:21.803112 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 13:10:21.805263 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:21.805271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:21.805359 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:21.805479 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:21.807239 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.808493 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.810296 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.811058 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.811205 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:21.814857 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:21.814866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:21.814874 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:21.814921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:21.818575 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:21.819392 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:21.819410 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:21.819417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:21.819593 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:21.820085 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:21.820092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:21.820099 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:21.820533 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:21.820692 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:21.821377 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 13:10:21.821680 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
I20260619 13:10:21.822840 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:21.801996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:44231)
0619 13:10:21.802925 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:21.802978 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:21.804980 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:21.805755 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:21.805763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:21.806271 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:21.807049 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:21.807068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.808688 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.808693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:21.810149 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:21.810162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:21.811268 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:21.811274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:21.811485 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:21.812209 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:21.812231 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:21.814609 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:21.818773 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:21.818788 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:21.818801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:21.819244 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:21.819766 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:21.819770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:21.819772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:21.819960 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:21.820655 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:21.820661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:21.820987 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 13:10:21.821218 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 13:10:21.821698 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":434,"threads_started":1}
I20260619 13:10:21.824456 14617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:21.824780 14617 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:21.825508 14617 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
I20260619 13:10:21.827093 14294 ts_manager.cc:194] Registered new tserver with Master: de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:10:21.828081 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:10:21.856317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:22.194303 14623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:22.194600 14623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:22.194757 14623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:22.202208 14623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:22.202330 14623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:22.202519 14623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:10:22.212913 14623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:22.214395 14623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:22.215794 14623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:22.229777 14628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:22.229741 14631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:22.229701 14629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:22.231006 14623 server_base.cc:1034] running on GCE node
I20260619 13:10:22.231648 14623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:22.232591 14623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:22.233785 14623 hybrid_clock.cc:648] HybridClock initialized: now 1781874622233730 us; error 31 us; skew 500 ppm
I20260619 13:10:22.234156 14623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:22 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:22.237793 14623 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:22.239357 14623 webserver.cc:460] Webserver started at http://127.0.97.3:38009/ using document root <none> and password file <none>
I20260619 13:10:22.240063 14623 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:22.240198 14623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:22.240505 14623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:22.243456 14623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "27aab526f3fa4befacf7db292a8eb66c"
format_stamp: "Formatted at 2026-06-19 13:10:22 on dist-test-slave-q11v"
server_key: "ba7ad42c7b23fe0a73254740f8cc833e"
server_key_iv: "d28ea185c006c4f1ed9ce8e19061abb3"
server_key_version: "encryptionkey@0"
I20260619 13:10:22.244247 14623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27aab526f3fa4befacf7db292a8eb66c"
format_stamp: "Formatted at 2026-06-19 13:10:22 on dist-test-slave-q11v"
server_key: "ba7ad42c7b23fe0a73254740f8cc833e"
server_key_iv: "d28ea185c006c4f1ed9ce8e19061abb3"
server_key_version: "encryptionkey@0"
I20260619 13:10:22.249495 14623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:10:22.253106 14638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:22.254542 14623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:10:22.254721 14623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "27aab526f3fa4befacf7db292a8eb66c"
format_stamp: "Formatted at 2026-06-19 13:10:22 on dist-test-slave-q11v"
server_key: "ba7ad42c7b23fe0a73254740f8cc833e"
server_key_iv: "d28ea185c006c4f1ed9ce8e19061abb3"
server_key_version: "encryptionkey@0"
I20260619 13:10:22.254930 14623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:22.288652 14623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:22.293109 14623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:22.293424 14623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:22.294533 14623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:22.295974 14623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:22.296052 14623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:22.296231 14623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:22.296283 14623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:22.305235 14483 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:22.330816 14623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33795
I20260619 13:10:22.330862 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33795 every 8 connection(s)
I20260619 13:10:22.332121 14623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:22.339110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14623
I20260619 13:10:22.339308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:10:22.339740 388 external_mini_cluster.cc:1428] Setting key 9050fe065109d420590f6d6ad2e6a914
Jun 19 13:10:22 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:22.353174 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.334790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42527 (local address 127.0.97.62:40361)
0619 13:10:22.335206 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.335210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.336261 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 13:10:22.338187 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.338193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.338287 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.338413 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:22.340051 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.340947 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.343003 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.343455 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.343620 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.347619 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:22.347625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:22.347628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:22.347669 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:22.350368 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.351071 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.351075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.351077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.351200 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.351610 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.351613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.351615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.351873 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:22.351986 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.352673 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.352848 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260619 13:10:22.354264 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.335407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:42527)
0619 13:10:22.336092 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.336139 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.337902 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.338652 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.338661 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.339211 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:22.339884 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.339903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.341106 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.341111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.342840 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.342853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.343757 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.343765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.343988 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.344624 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:22.344652 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:22.347385 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:22.350502 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:22.350509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:22.350522 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:22.350952 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:22.351298 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:22.351301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:22.351304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:22.351478 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:22.351976 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:22.351982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:22.352331 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.352563 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.352964 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260619 13:10:22.355726 14751 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:22.355991 14751 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:22.356662 14751 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
I20260619 13:10:22.358110 14293 ts_manager.cc:194] Registered new tserver with Master: 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795)
I20260619 13:10:22.359079 14293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42527
I20260619 13:10:22.367925 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:10:22.386135 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.372377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.97.62:40361)
0619 13:10:22.372548 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.372553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.372987 (+ 434us) server_negotiation.cc:407] Connection header received
0619 13:10:22.373136 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.373139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.373203 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.373286 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:22.374665 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.375394 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.376732 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.377113 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.377201 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.379680 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:22.379685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:22.379689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:22.379720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:22.383095 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.383768 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.383776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.383782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.383937 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.384307 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.384313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.384317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.384829 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:22.385125 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.385368 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.385684 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:10:22.390664 14293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:10:22.397660 14293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:10:22.431970 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.410470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33795 (local address 127.0.0.1:33470)
0619 13:10:22.422953 (+ 12483us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.422972 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.423196 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.424260 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.424268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.424315 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:22.424686 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.424700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.426820 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.426825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.428970 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.428982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.430950 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.430956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.431166 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.431221 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.431375 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.431575 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12370,"thread_start_us":12507,"threads_started":1}
I20260619 13:10:22.431973 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.410807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.97.3:33795)
0619 13:10:22.422252 (+ 11445us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.422267 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.423083 (+ 816us) server_negotiation.cc:407] Connection header received
0619 13:10:22.423383 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.423394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.423716 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.423908 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:22.424901 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.426550 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.429140 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.430741 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.430873 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.431123 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.431427 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.431618 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11182,"thread_start_us":11469,"threads_started":1}
I20260619 13:10:22.433261 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.410219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.0.1:40376)
0619 13:10:22.422357 (+ 12138us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.422429 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.422713 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.425519 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.425525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.425550 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:22.426033 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.426046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.427994 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.428000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.430149 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.430162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.432455 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.432462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.432642 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.432679 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.432763 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.432902 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11767,"thread_start_us":12473,"threads_started":1}
I20260619 13:10:22.433337 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.410269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.97.2:32771)
0619 13:10:22.424682 (+ 14413us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.424691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.424719 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:10:22.424826 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.424834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.425090 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.425285 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:22.426213 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.427857 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.430349 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.431841 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.431971 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.432447 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.432859 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.433017 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14249,"thread_start_us":14103,"threads_started":1}
I20260619 13:10:22.434553 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.424529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.97.1:37879)
0619 13:10:22.425218 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.425227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.425967 (+ 740us) server_negotiation.cc:407] Connection header received
0619 13:10:22.426291 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.426299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.426517 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.426677 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:22.427569 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.429181 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.431621 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.433108 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.433240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.433503 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.434066 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.434244 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":393,"threads_started":1}
I20260619 13:10:22.436224 14686 tablet_service.cc:1467] Processing CreateTablet for tablet aa35a61e15254a15910f9404fcdd9a5f (DEFAULT_TABLE table=test-table [id=217160f550e142048544b05186d9f587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:22.434475 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.425233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33390)
0619 13:10:22.425844 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.425866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.426078 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.426992 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.426999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.427023 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:22.427384 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.427392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.429320 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.429325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.431433 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.431447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.433676 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.433683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.433892 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.433925 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.434021 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.434147 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":301,"threads_started":1}
I20260619 13:10:22.437727 14686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35a61e15254a15910f9404fcdd9a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:22.440260 14552 tablet_service.cc:1467] Processing CreateTablet for tablet aa35a61e15254a15910f9404fcdd9a5f (DEFAULT_TABLE table=test-table [id=217160f550e142048544b05186d9f587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:22.440645 14418 tablet_service.cc:1467] Processing CreateTablet for tablet aa35a61e15254a15910f9404fcdd9a5f (DEFAULT_TABLE table=test-table [id=217160f550e142048544b05186d9f587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:22.441586 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35a61e15254a15910f9404fcdd9a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:22.442193 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35a61e15254a15910f9404fcdd9a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:22.453233 14770 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Bootstrap starting.
I20260619 13:10:22.457782 14770 tablet_bootstrap.cc:654] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:22.458138 14772 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Bootstrap starting.
I20260619 13:10:22.459282 14771 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Bootstrap starting.
I20260619 13:10:22.459326 14770 log.cc:824] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:22.462532 14772 tablet_bootstrap.cc:654] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:22.463740 14771 tablet_bootstrap.cc:654] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:22.463790 14770 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: No bootstrap required, opened a new log
I20260619 13:10:22.463882 14772 log.cc:824] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:22.464079 14770 ts_tablet_manager.cc:1397] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260619 13:10:22.465122 14771 log.cc:824] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:22.468182 14771 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: No bootstrap required, opened a new log
I20260619 13:10:22.468465 14771 ts_tablet_manager.cc:1397] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 13:10:22.469048 14772 tablet_bootstrap.cc:492] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: No bootstrap required, opened a new log
I20260619 13:10:22.469300 14772 ts_tablet_manager.cc:1397] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 13:10:22.474701 14770 raft_consensus.cc:348] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.475339 14770 raft_consensus.cc:374] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:22.475531 14770 raft_consensus.cc:729] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7031b9c04749bd823c462adfeb0e61, State: Initialized, Role: FOLLOWER
I20260619 13:10:22.476253 14770 consensus_queue.cc:260] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.477469 14617 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:22.478185 14770 ts_tablet_manager.cc:1428] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260619 13:10:22.479866 14771 raft_consensus.cc:348] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.480418 14771 raft_consensus.cc:374] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:22.480572 14771 raft_consensus.cc:729] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aab526f3fa4befacf7db292a8eb66c, State: Initialized, Role: FOLLOWER
I20260619 13:10:22.480350 14772 raft_consensus.cc:348] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.480810 14772 raft_consensus.cc:374] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:22.480908 14772 raft_consensus.cc:729] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ae5600cac2420ca7d5a77bc11abf72, State: Initialized, Role: FOLLOWER
I20260619 13:10:22.481287 14771 consensus_queue.cc:260] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.481503 14772 consensus_queue.cc:260] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.482971 14751 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:22.483460 14772 ts_tablet_manager.cc:1428] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260619 13:10:22.483572 14771 ts_tablet_manager.cc:1428] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260619 13:10:22.525828 14484 tablet.cc:2369] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:10:22.550794 14618 tablet.cc:2369] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:22.552307 14777 raft_consensus.cc:482] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:10:22.552645 14777 raft_consensus.cc:504] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.554447 14777 leader_election.cc:288] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771), 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879)
I20260619 13:10:22.561198 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.554823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48847 (local address 127.0.97.2:32771)
0619 13:10:22.555053 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.555060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.555388 (+ 328us) server_negotiation.cc:407] Connection header received
0619 13:10:22.555642 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.555649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.555734 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.555845 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:22.556308 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.557254 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.559177 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.560252 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.560357 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.560431 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.560799 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.560899 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:10:22.561223 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.554902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.97.3:48847)
0619 13:10:22.555245 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.555269 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.555450 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.555865 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.555869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.555918 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:22.556141 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.556153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.557390 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.557395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.559024 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.559036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.560410 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.560416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.560627 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.560664 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.560766 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.560907 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":3}
I20260619 13:10:22.561846 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.555036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57887 (local address 127.0.97.1:37879)
0619 13:10:22.555280 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.555287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.555924 (+ 637us) server_negotiation.cc:407] Connection header received
0619 13:10:22.556120 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.556127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.556208 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.556306 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:22.556865 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.557720 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.559813 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.560872 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.560992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.561108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.561511 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.561667 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:10:22.562065 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.554931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.97.3:57887)
0619 13:10:22.555808 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.555832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:22.555947 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:22.556323 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:22.556327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:22.556354 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:22.556702 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.556713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.557881 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.557885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.559662 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:22.559678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.561102 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.561108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:22.561300 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.561338 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:10:22.561475 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 13:10:22.561640 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":19,"thread_start_us":460,"threads_started":1}
I20260619 13:10:22.562595 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35a61e15254a15910f9404fcdd9a5f" candidate_uuid: "27aab526f3fa4befacf7db292a8eb66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7031b9c04749bd823c462adfeb0e61" is_pre_election: true
I20260619 13:10:22.563136 14572 raft_consensus.cc:2454] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aab526f3fa4befacf7db292a8eb66c in term 0.
I20260619 13:10:22.563083 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35a61e15254a15910f9404fcdd9a5f" candidate_uuid: "27aab526f3fa4befacf7db292a8eb66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ae5600cac2420ca7d5a77bc11abf72" is_pre_election: true
I20260619 13:10:22.563531 14438 raft_consensus.cc:2454] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aab526f3fa4befacf7db292a8eb66c in term 0.
I20260619 13:10:22.563925 14641 leader_election.cc:302] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aab526f3fa4befacf7db292a8eb66c, de7031b9c04749bd823c462adfeb0e61; no voters:
I20260619 13:10:22.564424 14777 raft_consensus.cc:2781] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:10:22.564607 14777 raft_consensus.cc:482] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:10:22.564718 14777 raft_consensus.cc:3037] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:22.567003 14777 raft_consensus.cc:504] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.567641 14777 leader_election.cc:288] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [CANDIDATE]: Term 1 election: Requested vote from peers de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771), 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879)
I20260619 13:10:22.568085 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35a61e15254a15910f9404fcdd9a5f" candidate_uuid: "27aab526f3fa4befacf7db292a8eb66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7031b9c04749bd823c462adfeb0e61"
I20260619 13:10:22.568161 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35a61e15254a15910f9404fcdd9a5f" candidate_uuid: "27aab526f3fa4befacf7db292a8eb66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ae5600cac2420ca7d5a77bc11abf72"
I20260619 13:10:22.568286 14572 raft_consensus.cc:3037] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:22.568414 14438 raft_consensus.cc:3037] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:22.570922 14572 raft_consensus.cc:2454] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aab526f3fa4befacf7db292a8eb66c in term 1.
I20260619 13:10:22.570922 14438 raft_consensus.cc:2454] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aab526f3fa4befacf7db292a8eb66c in term 1.
I20260619 13:10:22.571408 14641 leader_election.cc:302] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27aab526f3fa4befacf7db292a8eb66c, 51ae5600cac2420ca7d5a77bc11abf72, de7031b9c04749bd823c462adfeb0e61; no voters:
I20260619 13:10:22.571739 14777 raft_consensus.cc:2781] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:22.572209 14777 raft_consensus.cc:686] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 1 LEADER]: Becoming Leader. State: Replica: 27aab526f3fa4befacf7db292a8eb66c, State: Running, Role: LEADER
I20260619 13:10:22.572774 14777 consensus_queue.cc:237] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } }
I20260619 13:10:22.577447 14293 catalog_manager.cc:5495] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "27aab526f3fa4befacf7db292a8eb66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } health_report { overall_health: UNKNOWN } } }
W20260619 13:10:22.583818 14752 tablet.cc:2369] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:22.630594 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.623518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.97.3:33795)
0619 13:10:22.623776 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.623781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.623802 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:10:22.624017 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.624022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.624119 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.624262 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:10:22.624780 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.626138 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.628313 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.628758 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.628865 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.629369 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:10:22.630022 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:10:22.630150 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.630228 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.630299 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 13:10:22.638877 14572 raft_consensus.cc:1261] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Refusing update from remote peer 27aab526f3fa4befacf7db292a8eb66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:22.638875 14438 raft_consensus.cc:1261] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Refusing update from remote peer 27aab526f3fa4befacf7db292a8eb66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:22.639969 14777 consensus_queue.cc:1035] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:22.640271 14780 consensus_queue.cc:1035] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:22.652236 14784 mvcc.cc:205] Tried to move back new op lower bound from 7298558454318424064 to 7298558454062125056. Current Snapshot: MvccSnapshot[applied={T|T < 7298558454318424064}]
I20260619 13:10:22.673823 14294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:22.674126 14294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:22.677863 14294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Sending DeleteTablet for 3 replicas of tablet aa35a61e15254a15910f9404fcdd9a5f
I20260619 13:10:22.679032 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet aa35a61e15254a15910f9404fcdd9a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40376
I20260619 13:10:22.679040 14686 tablet_service.cc:1514] Processing DeleteTablet for tablet aa35a61e15254a15910f9404fcdd9a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33470
I20260619 13:10:22.681854 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet aa35a61e15254a15910f9404fcdd9a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33390
I20260619 13:10:22.690284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14355
I20260619 13:10:22.691560 14793 tablet_replica.cc:330] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: stopping tablet replica
I20260619 13:10:22.692516 14793 raft_consensus.cc:2229] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:22.693252 14793 raft_consensus.cc:2258] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:22.696341 14793 ts_tablet_manager.cc:1905] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:22.705217 14793 ts_tablet_manager.cc:1918] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:22.705533 14793 log.cc:1196] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/aa35a61e15254a15910f9404fcdd9a5f
Jun 19 13:10:22 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:10:22.705773 14795 tablet_replica.cc:330] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: stopping tablet replica
I20260619 13:10:22.706292 14793 ts_tablet_manager.cc:1939] T aa35a61e15254a15910f9404fcdd9a5f P de7031b9c04749bd823c462adfeb0e61: Deleting consensus metadata
I20260619 13:10:22.706432 14795 raft_consensus.cc:2229] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:22.707026 14795 raft_consensus.cc:2258] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:22.709318 14280 catalog_manager.cc:4841] TS de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771): tablet aa35a61e15254a15910f9404fcdd9a5f (table test-table [id=217160f550e142048544b05186d9f587]) successfully deleted
I20260619 13:10:22.711382 14795 ts_tablet_manager.cc:1905] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:22.712963 14794 tablet_replica.cc:330] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: stopping tablet replica
I20260619 13:10:22.713519 14794 raft_consensus.cc:2229] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:10:22.714360 14794 raft_consensus.cc:2258] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:22.714522 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.691840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.97.1:37879)
0619 13:10:22.692066 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.692075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.694459 (+ 2384us) server_negotiation.cc:407] Connection header received
0619 13:10:22.694677 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.694683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.694773 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.694891 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:22.696929 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.698153 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.700369 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.701062 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.701162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.707064 (+ 5902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:22.707083 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:22.707104 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:22.707157 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:22.710729 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.711670 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.711678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.711681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.711870 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.712351 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.712357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.712361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.712890 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:22.713213 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.713851 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.714105 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:10:22.716095 14478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:22.717233 14794 ts_tablet_manager.cc:1905] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:22.725798 14794 ts_tablet_manager.cc:1918] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:22.726084 14794 log.cc:1196] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/aa35a61e15254a15910f9404fcdd9a5f
I20260619 13:10:22.726788 14794 ts_tablet_manager.cc:1939] T aa35a61e15254a15910f9404fcdd9a5f P 27aab526f3fa4befacf7db292a8eb66c: Deleting consensus metadata
I20260619 13:10:22.729130 14279 catalog_manager.cc:4841] TS 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795): tablet aa35a61e15254a15910f9404fcdd9a5f (table test-table [id=217160f550e142048544b05186d9f587]) successfully deleted
I20260619 13:10:22.808950 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14355
I20260619 13:10:22.829799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14489
W20260619 13:10:22.829834 14279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37879 (error 108)
I20260619 13:10:22.831965 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.831367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33420)
0619 13:10:22.831663 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.831782 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260619 13:10:22.832230 14279 catalog_manager.cc:4584] TS 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa35a61e15254a15910f9404fcdd9a5f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Jun 19 13:10:22 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:10:22.847198 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.830968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.97.2:32771)
0619 13:10:22.831141 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.831147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.831167 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:10:22.831230 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.831236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.831316 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.831426 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:22.833040 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.834219 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.835570 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.836212 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.836334 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.840798 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:22.840811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:22.840826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:22.840864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:22.843925 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.844735 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.844743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.844745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.844880 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.845284 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.845288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.845291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.845987 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:22.846248 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.846661 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.846841 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:10:22.847931 14612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:22.886096 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.885482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33422)
0619 13:10:22.885764 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.885883 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 13:10:22.887351 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.886922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33436)
0619 13:10:22.887090 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.887180 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:10:22.938247 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14489
I20260619 13:10:22.948338 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.947741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33452)
0619 13:10:22.948018 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.948134 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 13:10:22.949651 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.949187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33454)
0619 13:10:22.949358 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:22.949470 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:10:22.956491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14623
Jun 19 13:10:22 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:10:22.972949 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:22.957303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.0.97.3:33795)
0619 13:10:22.957483 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:22.957490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:22.957737 (+ 247us) server_negotiation.cc:407] Connection header received
0619 13:10:22.957935 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:22.957941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:22.958008 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:22.958150 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:22.959447 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.960277 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.961675 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:22.962096 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:22.962242 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:22.966801 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:22.966814 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:22.966829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:22.966871 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:22.970014 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.970768 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.970775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.970777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.970918 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:22.971311 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:22.971317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:22.971320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:22.971731 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:22.972063 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:22.972351 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:10:22.972574 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:10:22.973906 14746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:23.030282 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:23.029788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33456)
0619 13:10:23.030005 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:23.030101 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:10:23.031422 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:23.031043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33472)
0619 13:10:23.031230 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:23.031316 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 13:10:23.096506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14623
I20260619 13:10:23.116340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14262
I20260619 13:10:23.117506 14324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:23.229033 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:23.228327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33486)
0619 13:10:23.228644 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:23.228763 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 13:10:23.229539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14262
I20260619 13:10:23.250965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--webserver_interface=127.0.97.62
--webserver_port=38457
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:23.581754 14802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:23.582096 14802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:23.582261 14802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:23.589365 14802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:23.589509 14802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:23.589663 14802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:23.589761 14802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:23.589840 14802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:23.599514 14802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:23.600864 14802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:23.602161 14802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:23.624454 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:23.624500 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:23.624496 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:23.625366 14802 server_base.cc:1034] running on GCE node
I20260619 13:10:23.626262 14802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:23.627854 14802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:23.629159 14802 hybrid_clock.cc:648] HybridClock initialized: now 1781874623629058 us; error 54 us; skew 500 ppm
I20260619 13:10:23.629752 14802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:23 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:23.633983 14802 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:23.635600 14802 webserver.cc:460] Webserver started at http://127.0.97.62:38457/ using document root <none> and password file <none>
I20260619 13:10:23.636351 14802 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:23.636513 14802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:23.642649 14802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:10:23.645671 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:23.647060 14802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:23.647258 14802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f0856d15dabe4c93b33dde41f46708b6"
format_stamp: "Formatted at 2026-06-19 13:10:20 on dist-test-slave-q11v"
server_key: "cbe6be9b5899d435ff36364aedf3f82b"
server_key_iv: "f257223e21252f10cb0dd3fd28ede3b4"
server_key_version: "encryptionkey@0"
I20260619 13:10:23.648376 14802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:23.662436 14802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:23.667080 14802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:23.667380 14802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:23.686795 14802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40361
I20260619 13:10:23.686803 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40361 every 8 connection(s)
I20260619 13:10:23.688369 14802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:23.692859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14802
I20260619 13:10:23.693776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37879
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=38589
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:23.695564 14870 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:10:23.698899 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap starting.
I20260619 13:10:23.728163 14870 log.cc:824] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:23.743803 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:10:23.744488 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap complete.
I20260619 13:10:23.758671 14870 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:23.760057 14870 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Initialized, Role: FOLLOWER
I20260619 13:10:23.761053 14870 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:23.761447 14870 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:23.761626 14870 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:23.761860 14870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:10:23.768185 14870 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:23.768816 14870 leader_election.cc:302] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0856d15dabe4c93b33dde41f46708b6; no voters:
I20260619 13:10:23.769508 14870 leader_election.cc:288] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:10:23.769831 14874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:10:23.771157 14874 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 LEADER]: Becoming Leader. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Running, Role: LEADER
I20260619 13:10:23.771780 14870 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:23.771775 14874 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:23.775058 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0856d15dabe4c93b33dde41f46708b6. Latest consensus state: current_term: 2 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:23.775866 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:23.775399 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:23.776597 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:23.787242 14881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:23.796154 14881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:23.797901 14881 catalog_manager.cc:1259] Loaded cluster ID: 4a1de8d14ee64af4bf4f00b3f2529592
I20260619 13:10:23.798063 14881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:23.801578 14881 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 13:10:23.819787 14881 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 13:10:23.839831 14881 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:23.841720 14881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Loaded TSK: 0
I20260619 13:10:23.843307 14881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:10:24.079856 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:24.080199 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:24.080368 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:24.088023 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:24.088192 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:24.088382 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:24.098877 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38589
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:24.100260 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:24.101764 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:24.115240 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:24.115407 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:24.115491 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:24.116479 14872 server_base.cc:1034] running on GCE node
I20260619 13:10:24.117141 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:24.118265 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:24.119494 14872 hybrid_clock.cc:648] HybridClock initialized: now 1781874624119448 us; error 41 us; skew 500 ppm
I20260619 13:10:24.119829 14872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:24 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:24.123270 14872 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:24.124855 14872 webserver.cc:460] Webserver started at http://127.0.97.1:38589/ using document root <none> and password file <none>
I20260619 13:10:24.125564 14872 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:24.125752 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:24.131510 14872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:10:24.134778 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:24.136226 14872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:10:24.136422 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "51ae5600cac2420ca7d5a77bc11abf72"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "c2efed587600c9244088af4d9d40933e"
server_key_iv: "1ea78841296dc5d10f7cb18c154e093b"
server_key_version: "encryptionkey@0"
I20260619 13:10:24.137416 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:24.158420 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:24.162819 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:24.163113 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:24.164143 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:24.166304 14914 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 13:10:24.172281 14914 ts_tablet_manager.cc:1852] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 13:10:24.173455 14914 ts_tablet_manager.cc:1905] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:24.180984 14914 ts_tablet_manager.cc:1918] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:24.181212 14914 log.cc:1196] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/aa35a61e15254a15910f9404fcdd9a5f
I20260619 13:10:24.181752 14914 ts_tablet_manager.cc:1939] T aa35a61e15254a15910f9404fcdd9a5f P 51ae5600cac2420ca7d5a77bc11abf72: Deleting consensus metadata
I20260619 13:10:24.182199 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 13:10:24.182330 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260619 13:10:24.182477 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:24.182570 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:24.217327 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37879
I20260619 13:10:24.217379 15020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37879 every 8 connection(s)
I20260619 13:10:24.218726 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:24.228571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 14872
I20260619 13:10:24.229213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32771
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=34087
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:10:24 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:24.243448 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:24.222003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:50419)
0619 13:10:24.222818 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:24.222865 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:24.224615 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:24.226060 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:24.226068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:24.226609 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:24.227140 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:24.227156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.228841 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.228845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:24.230302 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:24.230315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.231380 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.231388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:24.231656 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:24.232393 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:24.232416 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:24.234950 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:24.238720 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:24.238730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:24.238747 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:24.239197 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:24.239672 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:24.239677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:24.239680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:24.239875 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:24.240688 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:24.240697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:24.241107 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 13:10:24.241466 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 13:10:24.242072 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":357,"threads_started":1}
I20260619 13:10:24.243443 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:24.221885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50419 (local address 127.0.97.62:40361)
0619 13:10:24.222608 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:24.222624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:24.223003 (+ 379us) server_negotiation.cc:407] Connection header received
0619 13:10:24.225166 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:24.225181 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:24.225543 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:24.225950 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:24.227365 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.228666 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.230461 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.231116 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.231334 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:24.235257 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:24.235265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:24.235285 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:24.235331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:24.238546 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:24.239356 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:24.239364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:24.239371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:24.239542 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:24.239999 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:24.240008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:24.240010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:24.240550 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:24.240760 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:24.241708 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 13:10:24.242142 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":263,"threads_started":1}
I20260619 13:10:24.246415 15021 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:24.246742 15021 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:24.247453 15021 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
I20260619 13:10:24.249711 14835 ts_manager.cc:194] Registered new tserver with Master: 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879)
I20260619 13:10:24.251158 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50419
W20260619 13:10:24.563880 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:24.564241 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:24.564419 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:24.571996 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:24.572165 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:24.572348 15025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:24.582583 15025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34087
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:24.583969 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:24.585295 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:24.598634 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:24.598714 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:24.598721 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:24.599740 15025 server_base.cc:1034] running on GCE node
I20260619 13:10:24.600394 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:24.601293 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:24.602497 15025 hybrid_clock.cc:648] HybridClock initialized: now 1781874624602416 us; error 68 us; skew 500 ppm
I20260619 13:10:24.602829 15025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:24 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:24.606074 15025 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:24.607445 15025 webserver.cc:460] Webserver started at http://127.0.97.2:34087/ using document root <none> and password file <none>
I20260619 13:10:24.608101 15025 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:24.608212 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:24.613651 15025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:24.616292 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:24.617736 15025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:10:24.617910 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "de7031b9c04749bd823c462adfeb0e61"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "12e153ba753ab5160a08413f14f2d7c0"
server_key_iv: "ff13d972a95643b82912bd74897ffd71"
server_key_version: "encryptionkey@0"
I20260619 13:10:24.618892 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:24.642771 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:24.647303 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:24.647586 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:24.648567 15025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:24.649966 15025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:24.650094 15025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:24.650223 15025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:24.650312 15025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:24.683959 15025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32771
I20260619 13:10:24.684053 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32771 every 8 connection(s)
I20260619 13:10:24.685433 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:24.693501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15025
I20260619 13:10:24.694260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33795
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=38009
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:10:24 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:24.706885 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:24.688322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39591 (local address 127.0.97.62:40361)
0619 13:10:24.688498 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:24.688503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:24.689870 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 13:10:24.691716 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:24.691722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:24.691817 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:24.691928 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:24.693477 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.694367 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.696070 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.696524 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.696662 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:24.700268 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:24.700274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:24.700277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:24.700311 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:24.703108 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:24.704112 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:24.704118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:24.704122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:24.704275 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:24.704798 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:24.704803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:24.704806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:24.705189 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:24.705348 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:24.706276 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 13:10:24.706529 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:10:24.708290 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:24.688896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:39591)
0619 13:10:24.689715 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:24.689753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:24.691443 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:24.692190 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:24.692198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:24.692724 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:24.693307 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:24.693324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.694582 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.694590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:24.695901 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:24.695913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:24.696776 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:24.696784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:24.697011 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:24.697772 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:24.697800 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:24.700038 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:24.703264 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:24.703277 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:24.703297 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:24.703958 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:24.704409 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:24.704414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:24.704419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:24.704679 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:24.705336 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:24.705347 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:24.705833 (+ 486us) client_negotiation.cc:769] Sending connection context
0619 13:10:24.706152 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 13:10:24.706735 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":307,"threads_started":1}
I20260619 13:10:24.709887 15153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:24.710242 15153 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:24.711177 15153 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
I20260619 13:10:24.712875 14835 ts_manager.cc:194] Registered new tserver with Master: de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771)
I20260619 13:10:24.713852 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39591
W20260619 13:10:25.035404 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:25.035725 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:25.035900 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:25.043701 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:25.043848 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:25.044144 15156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:10:25.055094 15156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38009
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:25.056542 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:25.058009 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:25.072065 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:25.072175 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:25.072121 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:25.073664 15156 server_base.cc:1034] running on GCE node
I20260619 13:10:25.074338 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:25.075343 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:25.076584 15156 hybrid_clock.cc:648] HybridClock initialized: now 1781874625076540 us; error 33 us; skew 500 ppm
I20260619 13:10:25.076944 15156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:25 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:25.080264 15156 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:25.081911 15156 webserver.cc:460] Webserver started at http://127.0.97.3:38009/ using document root <none> and password file <none>
I20260619 13:10:25.082576 15156 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:25.082695 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:25.088321 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 13:10:25.091274 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:25.092815 15156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:10:25.092998 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "27aab526f3fa4befacf7db292a8eb66c"
format_stamp: "Formatted at 2026-06-19 13:10:22 on dist-test-slave-q11v"
server_key: "ba7ad42c7b23fe0a73254740f8cc833e"
server_key_iv: "d28ea185c006c4f1ed9ce8e19061abb3"
server_key_version: "encryptionkey@0"
I20260619 13:10:25.094084 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:25.123001 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:25.127683 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:25.127975 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:25.128953 15156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:25.130391 15156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:25.130513 15156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:25.130652 15156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:25.130761 15156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:25.164364 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33795
I20260619 13:10:25.164444 15283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33795 every 8 connection(s)
I20260619 13:10:25.166016 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:25.173697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15156
Jun 19 13:10:25 dist-test-slave-q11v krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:25.188700 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.168539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52891 (local address 127.0.97.62:40361)
0619 13:10:25.168766 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.168772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.170110 (+ 1338us) server_negotiation.cc:407] Connection header received
0619 13:10:25.171986 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.171990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.172061 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.172180 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:25.173894 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.174989 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.177463 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.178140 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.178256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.182302 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:25.182306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:25.182310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:25.182338 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:25.185522 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.186289 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.186294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.186298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.186440 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.186870 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.186875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.186878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.187216 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:25.187381 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.188128 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.188340 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:10:25.189977 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.169184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:52891)
0619 13:10:25.169953 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.169991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.171701 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.172418 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.172428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.173060 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:25.173701 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.173720 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.175210 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.175218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.177312 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.177340 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.178352 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.178357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.178558 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.179240 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:25.179270 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:25.182105 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:25.185716 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:25.185727 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:25.185745 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:25.186161 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:25.186552 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:25.186558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:25.186562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:25.186748 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:25.187345 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:25.187353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:25.187743 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.188006 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.188536 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":323,"threads_started":1}
I20260619 13:10:25.191262 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.174588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.97.62:40361)
0619 13:10:25.175125 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.175131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.175689 (+ 558us) server_negotiation.cc:407] Connection header received
0619 13:10:25.175841 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.175844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.175898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.175979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:25.177817 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.178739 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.180689 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.181382 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.181520 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.185490 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:25.185494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:25.185498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:25.185527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:25.188181 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.189020 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.189025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.189029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.189161 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.189518 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.189522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.189525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.189879 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:25.190147 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.190714 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.190906 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":276,"threads_started":1}
I20260619 13:10:25.191581 15284 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40361
I20260619 13:10:25.191920 15284 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:25.192741 15284 heartbeater.cc:511] Master 127.0.97.62:40361 requested a full tablet report, sending...
I20260619 13:10:25.194221 14835 ts_manager.cc:194] Registered new tserver with Master: 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795)
I20260619 13:10:25.195087 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52891
I20260619 13:10:25.206238 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:10:25.212852 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.207243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.0.97.62:40361)
0619 13:10:25.207497 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.207503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.207719 (+ 216us) server_negotiation.cc:407] Connection header received
0619 13:10:25.207905 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.207909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.208077 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.208248 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:10:25.208607 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.209303 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.210767 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.211208 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.211301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.211724 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:10:25.212274 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:10:25.212380 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.212472 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.212542 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 13:10:25.213927 14835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:10:25.221372 14835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:10:25.244793 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.233153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33496)
0619 13:10:25.235540 (+ 2387us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.235554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.235701 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.236299 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.236307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.236335 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:25.236836 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.236854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.238850 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.238856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.241552 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.241565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.243814 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.243820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.244024 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.244078 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.244203 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.244391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2294,"thread_start_us":2106,"threads_started":1}
I20260619 13:10:25.245570 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.233804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.97.1:37879)
0619 13:10:25.234370 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.234384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.235836 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 13:10:25.235928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.235933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.236103 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.236285 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:25.237094 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.238686 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.241757 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.243267 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.243394 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.243675 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.245041 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.245217 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":998,"threads_started":1}
I20260619 13:10:25.247140 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.232634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.0.1:40384)
0619 13:10:25.234407 (+ 1773us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.234458 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.234695 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.236555 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.236559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.236580 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:25.236988 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.237000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.239980 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.239986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.243395 (+ 3409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.243408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.246312 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.246318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.246500 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.246532 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.246626 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.246757 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"mutex_wait_us":3,"thread_start_us":1357,"threads_started":1}
I20260619 13:10:25.247226 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.233054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.0.97.2:32771)
0619 13:10:25.233746 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.233755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.234595 (+ 840us) server_negotiation.cc:407] Connection header received
0619 13:10:25.235019 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.235033 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.235342 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.235513 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:25.237186 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.239797 (+ 2611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.243569 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.245019 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.246173 (+ 1154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.246319 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.246698 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.246882 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":349,"threads_started":1}
I20260619 13:10:25.248124 14956 tablet_service.cc:1467] Processing CreateTablet for tablet a9dfcae3c65b4853a16ca26359341c02 (DEFAULT_TABLE table=test-table [id=3c4eef1297554a2888e016143afc8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:25.249624 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dfcae3c65b4853a16ca26359341c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:25.250936 15088 tablet_service.cc:1467] Processing CreateTablet for tablet a9dfcae3c65b4853a16ca26359341c02 (DEFAULT_TABLE table=test-table [id=3c4eef1297554a2888e016143afc8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:25.252816 15088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dfcae3c65b4853a16ca26359341c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:25.254058 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.236115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.0.97.3:33795)
0619 13:10:25.245658 (+ 9543us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.245668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.246383 (+ 715us) server_negotiation.cc:407] Connection header received
0619 13:10:25.246468 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.246474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.246716 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.246918 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:25.247779 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.249172 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.251430 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.252882 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.253014 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.253322 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.253505 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.253701 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9322,"thread_start_us":9131,"threads_started":1}
I20260619 13:10:25.254108 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.235683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33795 (local address 127.0.0.1:33500)
0619 13:10:25.245902 (+ 10219us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.245920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.246051 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.247197 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.247204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.247234 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:25.247595 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.247604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.249322 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.249327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.251259 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.251273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.253094 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.253100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.253319 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.253352 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.253442 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.253730 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10088,"thread_start_us":10099,"threads_started":1}
I20260619 13:10:25.254292 15021 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:25.258021 15219 tablet_service.cc:1467] Processing CreateTablet for tablet a9dfcae3c65b4853a16ca26359341c02 (DEFAULT_TABLE table=test-table [id=3c4eef1297554a2888e016143afc8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:25.259414 15219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dfcae3c65b4853a16ca26359341c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:25.260759 15300 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Bootstrap starting.
I20260619 13:10:25.264436 15300 tablet_bootstrap.cc:654] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:25.265681 15300 log.cc:824] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:25.268844 15300 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: No bootstrap required, opened a new log
I20260619 13:10:25.268973 15302 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Bootstrap starting.
I20260619 13:10:25.269222 15300 ts_tablet_manager.cc:1397] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:10:25.272284 15302 tablet_bootstrap.cc:654] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:25.273469 15302 log.cc:824] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:25.273732 15303 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Bootstrap starting.
I20260619 13:10:25.278298 15303 tablet_bootstrap.cc:654] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:25.278802 15300 raft_consensus.cc:348] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.279168 15300 raft_consensus.cc:374] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:25.279302 15300 raft_consensus.cc:729] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ae5600cac2420ca7d5a77bc11abf72, State: Initialized, Role: FOLLOWER
I20260619 13:10:25.279595 15303 log.cc:824] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:25.279938 15300 consensus_queue.cc:260] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.280418 15302 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: No bootstrap required, opened a new log
I20260619 13:10:25.280731 15302 ts_tablet_manager.cc:1397] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260619 13:10:25.281661 15300 ts_tablet_manager.cc:1428] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 13:10:25.282711 15303 tablet_bootstrap.cc:492] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: No bootstrap required, opened a new log
I20260619 13:10:25.282958 15303 ts_tablet_manager.cc:1397] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260619 13:10:25.292093 15302 raft_consensus.cc:348] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.292593 15302 raft_consensus.cc:374] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:25.292706 15302 raft_consensus.cc:729] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7031b9c04749bd823c462adfeb0e61, State: Initialized, Role: FOLLOWER
I20260619 13:10:25.293377 15302 consensus_queue.cc:260] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.293378 15303 raft_consensus.cc:348] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.293843 15303 raft_consensus.cc:374] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:25.293967 15303 raft_consensus.cc:729] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aab526f3fa4befacf7db292a8eb66c, State: Initialized, Role: FOLLOWER
I20260619 13:10:25.294700 15153 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:25.294598 15303 consensus_queue.cc:260] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.295444 15302 ts_tablet_manager.cc:1428] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 13:10:25.295642 15284 heartbeater.cc:503] Master 127.0.97.62:40361 was elected leader, sending a full tablet report...
I20260619 13:10:25.297514 15303 ts_tablet_manager.cc:1428] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260619 13:10:25.351729 15306 raft_consensus.cc:482] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:10:25.352067 15306 raft_consensus.cc:504] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.354108 15306 leader_election.cc:288] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771), 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795)
I20260619 13:10:25.361898 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.354607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56871 (local address 127.0.97.3:33795)
0619 13:10:25.354865 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.354874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.355557 (+ 683us) server_negotiation.cc:407] Connection header received
0619 13:10:25.355743 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.355746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.355804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.355903 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:25.356572 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.357292 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.359750 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.360998 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.361138 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.361233 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.361482 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.361643 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 13:10:25.362021 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.354477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33795 (local address 127.0.97.1:56871)
0619 13:10:25.355442 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.355457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.355606 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.355918 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.355922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.355976 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:25.356397 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.356406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.357449 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.357456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.359512 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.359535 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.361191 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.361195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.361320 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.361347 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.361427 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.361523 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":52,"spinlock_wait_cycles":33152,"thread_start_us":476,"threads_started":1}
I20260619 13:10:25.362211 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.354492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38317 (local address 127.0.97.2:32771)
0619 13:10:25.354735 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.354745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.355504 (+ 759us) server_negotiation.cc:407] Connection header received
0619 13:10:25.355740 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.355744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.355804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.355904 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:25.356503 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.357222 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.359748 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.360994 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.361130 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.361227 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.361804 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.361945 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:10:25.362360 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.354443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32771 (local address 127.0.97.1:38317)
0619 13:10:25.355386 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.355402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:25.355578 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:25.355918 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:25.355922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:25.355944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:25.356347 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.356357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.357382 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.357390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.359511 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:25.359534 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.361387 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.361391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:25.361578 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.361648 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 13:10:25.361741 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 13:10:25.361865 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":3,"thread_start_us":685,"threads_started":1}
I20260619 13:10:25.363348 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dfcae3c65b4853a16ca26359341c02" candidate_uuid: "51ae5600cac2420ca7d5a77bc11abf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7031b9c04749bd823c462adfeb0e61" is_pre_election: true
I20260619 13:10:25.363348 15239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dfcae3c65b4853a16ca26359341c02" candidate_uuid: "51ae5600cac2420ca7d5a77bc11abf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aab526f3fa4befacf7db292a8eb66c" is_pre_election: true
I20260619 13:10:25.363831 15108 raft_consensus.cc:2454] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ae5600cac2420ca7d5a77bc11abf72 in term 0.
I20260619 13:10:25.363842 15239 raft_consensus.cc:2454] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ae5600cac2420ca7d5a77bc11abf72 in term 0.
I20260619 13:10:25.364481 14910 leader_election.cc:302] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ae5600cac2420ca7d5a77bc11abf72, de7031b9c04749bd823c462adfeb0e61; no voters:
I20260619 13:10:25.364885 15306 raft_consensus.cc:2781] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:10:25.365036 15306 raft_consensus.cc:482] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:10:25.365136 15306 raft_consensus.cc:3037] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:25.367385 15306 raft_consensus.cc:504] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.368021 15306 leader_election.cc:288] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [CANDIDATE]: Term 1 election: Requested vote from peers de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771), 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795)
I20260619 13:10:25.368404 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dfcae3c65b4853a16ca26359341c02" candidate_uuid: "51ae5600cac2420ca7d5a77bc11abf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7031b9c04749bd823c462adfeb0e61"
I20260619 13:10:25.368516 15239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dfcae3c65b4853a16ca26359341c02" candidate_uuid: "51ae5600cac2420ca7d5a77bc11abf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aab526f3fa4befacf7db292a8eb66c"
I20260619 13:10:25.368613 15108 raft_consensus.cc:3037] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:25.368754 15239 raft_consensus.cc:3037] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:25.371207 15239 raft_consensus.cc:2454] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ae5600cac2420ca7d5a77bc11abf72 in term 1.
I20260619 13:10:25.371207 15108 raft_consensus.cc:2454] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ae5600cac2420ca7d5a77bc11abf72 in term 1.
I20260619 13:10:25.371651 14909 leader_election.cc:302] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aab526f3fa4befacf7db292a8eb66c, 51ae5600cac2420ca7d5a77bc11abf72; no voters:
I20260619 13:10:25.371984 15306 raft_consensus.cc:2781] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:25.372490 15306 raft_consensus.cc:686] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 LEADER]: Becoming Leader. State: Replica: 51ae5600cac2420ca7d5a77bc11abf72, State: Running, Role: LEADER
I20260619 13:10:25.372973 15306 consensus_queue.cc:237] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } }
I20260619 13:10:25.377895 14834 catalog_manager.cc:5495] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "51ae5600cac2420ca7d5a77bc11abf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ae5600cac2420ca7d5a77bc11abf72" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 } health_report { overall_health: UNKNOWN } } }
I20260619 13:10:25.402735 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.397471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.97.1:37879)
0619 13:10:25.397702 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.397706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.397741 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:10:25.397940 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.397944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.398033 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.398172 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:10:25.398564 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.399300 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.400701 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.401084 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.401181 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.401701 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:10:25.402242 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:10:25.402354 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.402465 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.402529 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:10:25.410872 15239 raft_consensus.cc:1261] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Refusing update from remote peer 51ae5600cac2420ca7d5a77bc11abf72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:25.410924 15108 raft_consensus.cc:1261] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Refusing update from remote peer 51ae5600cac2420ca7d5a77bc11abf72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:25.411866 15311 consensus_queue.cc:1035] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7031b9c04749bd823c462adfeb0e61" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:25.412216 15306 consensus_queue.cc:1035] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aab526f3fa4befacf7db292a8eb66c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:10:25.417582 15285 tablet.cc:2369] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:25.437410 14834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:25.437760 14834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:25.441453 14834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Sending DeleteTablet for 3 replicas of tablet a9dfcae3c65b4853a16ca26359341c02
I20260619 13:10:25.442615 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet a9dfcae3c65b4853a16ca26359341c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40384
I20260619 13:10:25.442839 15219 tablet_service.cc:1514] Processing DeleteTablet for tablet a9dfcae3c65b4853a16ca26359341c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33500
I20260619 13:10:25.442845 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet a9dfcae3c65b4853a16ca26359341c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33496
I20260619 13:10:25.444219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14872
I20260619 13:10:25.454993 15323 tablet_replica.cc:330] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: stopping tablet replica
I20260619 13:10:25.455705 15323 raft_consensus.cc:2229] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:25.456372 15323 raft_consensus.cc:2258] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:25.458766 15323 ts_tablet_manager.cc:1905] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:25.466109 15323 ts_tablet_manager.cc:1918] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:25.466367 15323 log.cc:1196] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/a9dfcae3c65b4853a16ca26359341c02
I20260619 13:10:25.467015 15323 ts_tablet_manager.cc:1939] T a9dfcae3c65b4853a16ca26359341c02 P de7031b9c04749bd823c462adfeb0e61: Deleting consensus metadata
I20260619 13:10:25.469151 14821 catalog_manager.cc:4841] TS de7031b9c04749bd823c462adfeb0e61 (127.0.97.2:32771): tablet a9dfcae3c65b4853a16ca26359341c02 (table test-table [id=3c4eef1297554a2888e016143afc8a53]) successfully deleted
I20260619 13:10:25.475172 15324 tablet_replica.cc:330] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: stopping tablet replica
I20260619 13:10:25.475654 15324 raft_consensus.cc:2229] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:25.476210 15324 raft_consensus.cc:2258] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:25.477036 15325 tablet_replica.cc:330] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: stopping tablet replica
I20260619 13:10:25.477525 15325 raft_consensus.cc:2229] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:10:25.478397 15325 raft_consensus.cc:2258] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:25.479007 15324 ts_tablet_manager.cc:1905] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:25.480191 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.446191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.0.97.1:37879)
0619 13:10:25.446426 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.446435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.446457 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:10:25.446540 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.446551 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.446650 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.446778 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:25.449465 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.451119 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.455265 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.456217 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.456384 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.466694 (+ 10310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:25.466720 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:25.466747 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:25.466825 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:25.473268 (+ 6443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.474398 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.474411 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.474418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.474779 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.477274 (+ 2495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.477281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.477284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.477879 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:25.478272 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.479523 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.479794 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:10:25.480336 15325 ts_tablet_manager.cc:1905] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:25.481030 15016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:25.486860 15324 ts_tablet_manager.cc:1918] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:25.487035 15324 log.cc:1196] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/a9dfcae3c65b4853a16ca26359341c02
I20260619 13:10:25.487584 15324 ts_tablet_manager.cc:1939] T a9dfcae3c65b4853a16ca26359341c02 P 27aab526f3fa4befacf7db292a8eb66c: Deleting consensus metadata
I20260619 13:10:25.489168 14820 catalog_manager.cc:4841] TS 27aab526f3fa4befacf7db292a8eb66c (127.0.97.3:33795): tablet a9dfcae3c65b4853a16ca26359341c02 (table test-table [id=3c4eef1297554a2888e016143afc8a53]) successfully deleted
I20260619 13:10:25.585179 15325 ts_tablet_manager.cc:1918] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:25.585378 15325 log.cc:1196] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/a9dfcae3c65b4853a16ca26359341c02
I20260619 13:10:25.585677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14872
W20260619 13:10:25.603686 14820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37879 (error 108)
I20260619 13:10:25.604400 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15025
I20260619 13:10:25.605659 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.605090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33526)
0619 13:10:25.605286 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.605409 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:10:25.605927 14820 catalog_manager.cc:4584] TS 51ae5600cac2420ca7d5a77bc11abf72 (127.0.97.1:37879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9dfcae3c65b4853a16ca26359341c02: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
I20260619 13:10:25.626164 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.605237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.2:32771)
0619 13:10:25.605404 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.605411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.605506 (+ 95us) server_negotiation.cc:407] Connection header received
0619 13:10:25.605690 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.605697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.605792 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.605899 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:25.607283 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.608042 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.609419 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.610001 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.610137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.620309 (+ 10172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:25.620327 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:25.620343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:25.620386 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:25.623324 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.624058 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.624065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.624067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.624189 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.624565 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.624572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.624575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.624964 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:25.625260 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.625576 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.625829 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 13:10:25.626875 15148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:25.659703 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.659119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33530)
0619 13:10:25.659362 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.659493 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:10:25.660782 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.660386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33532)
0619 13:10:25.660539 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.660622 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:10:25.719630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15025
I20260619 13:10:25.722442 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.721954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33542)
0619 13:10:25.722161 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.722259 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:10:25.723495 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.723148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33554)
0619 13:10:25.723299 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.723377 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 13:10:25.734513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15156
I20260619 13:10:25.747548 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.735274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.0.97.3:33795)
0619 13:10:25.735458 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:25.735463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:25.735482 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:10:25.735622 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:25.735626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:25.735712 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:25.735796 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:25.737075 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.737866 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.739096 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:25.739470 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:25.739558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:25.742216 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:25.742230 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:25.742246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:25.742277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:25.745005 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.745759 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.745766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.745769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.745936 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:25.746290 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:25.746293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:25.746295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:25.746618 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:25.746847 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:25.747034 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 13:10:25.747210 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:10:25.748519 15279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:25.804250 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.803752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33562)
0619 13:10:25.803979 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.804092 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 13:10:25.805369 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:25.804962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37879 (local address 127.0.0.1:33568)
0619 13:10:25.805187 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:25.805261 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 13:10:25.838424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15156
I20260619 13:10:25.855078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 14802
I20260619 13:10:25.856206 14865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:25.972632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 14802
I20260619 13:10:25.994622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--webserver_interface=127.0.97.62
--webserver_port=38457
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:26.335721 15331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:26.336009 15331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:26.336185 15331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:26.343374 15331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:26.343484 15331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:26.343530 15331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:26.343580 15331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:26.343672 15331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:26.353744 15331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:26.355146 15331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:26.356678 15331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:26.368820 15337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:26.368863 15336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:26.368933 15339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:26.369807 15331 server_base.cc:1034] running on GCE node
I20260619 13:10:26.370754 15331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:26.372180 15331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:26.373389 15331 hybrid_clock.cc:648] HybridClock initialized: now 1781874626373346 us; error 33 us; skew 500 ppm
I20260619 13:10:26.373764 15331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:26 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:26.376893 15331 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:26.378516 15331 webserver.cc:460] Webserver started at http://127.0.97.62:38457/ using document root <none> and password file <none>
I20260619 13:10:26.379181 15331 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:26.379283 15331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:26.384771 15331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:10:26.387743 15346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:26.389427 15331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:26.389662 15331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f0856d15dabe4c93b33dde41f46708b6"
format_stamp: "Formatted at 2026-06-19 13:10:20 on dist-test-slave-q11v"
server_key: "cbe6be9b5899d435ff36364aedf3f82b"
server_key_iv: "f257223e21252f10cb0dd3fd28ede3b4"
server_key_version: "encryptionkey@0"
I20260619 13:10:26.390702 15331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:26.414839 15331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:26.419490 15331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:26.419719 15331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:26.439385 15331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40361
I20260619 13:10:26.439419 15397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40361 every 8 connection(s)
I20260619 13:10:26.440956 15331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:26.447716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15331
I20260619 13:10:26.448163 15398 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:10:26.448794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37879
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=38589
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:26.453543 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap starting.
I20260619 13:10:26.475153 15398 log.cc:824] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:26.491660 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:10:26.492230 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6: Bootstrap complete.
I20260619 13:10:26.502022 15398 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:26.503012 15398 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Initialized, Role: FOLLOWER
I20260619 13:10:26.503705 15398 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:26.503971 15398 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:26.504108 15398 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:26.504288 15398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:10:26.508212 15398 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:26.508728 15398 leader_election.cc:302] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0856d15dabe4c93b33dde41f46708b6; no voters:
I20260619 13:10:26.509433 15398 leader_election.cc:288] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:10:26.509583 15402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:10:26.510824 15402 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [term 3 LEADER]: Becoming Leader. State: Replica: f0856d15dabe4c93b33dde41f46708b6, State: Running, Role: LEADER
I20260619 13:10:26.511427 15402 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } }
I20260619 13:10:26.511723 15398 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:26.513953 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0856d15dabe4c93b33dde41f46708b6. Latest consensus state: current_term: 3 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:26.514065 15403 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f0856d15dabe4c93b33dde41f46708b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0856d15dabe4c93b33dde41f46708b6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40361 } } }
I20260619 13:10:26.514585 15403 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:26.514916 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0856d15dabe4c93b33dde41f46708b6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:26.521075 15409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:26.530694 15409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:26.532205 15409 catalog_manager.cc:1259] Loaded cluster ID: 4a1de8d14ee64af4bf4f00b3f2529592
I20260619 13:10:26.532371 15409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 13:10:26.534468 15409 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7eb50c19ad google::LogMessage::Fail() at ??:0
@ 0x7f7eb50c488a google::LogMessage::SendToLog() at ??:0
@ 0x7f7eb50c14de google::LogMessage::Flush() at ??:0
@ 0x7f7eb50c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ecaaf4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7ecaaf1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7ecab79610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7ecab79439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7eb7d0d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7eb7d1091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7eb7d10799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7eb7cebeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ebafc56db start_thread at ??:0
@ 0x7f7eb3f5b61f clone at ??:0
W20260619 13:10:26.854887 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:26.855159 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:26.855254 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:26.862701 15400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:26.862823 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:26.862947 15400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:26.873016 15400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38589
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:26.874384 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:26.875676 15400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:26.888698 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:26.888772 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:26.888792 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:26.889470 15400 server_base.cc:1034] running on GCE node
I20260619 13:10:26.890141 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:26.891052 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:26.892352 15400 hybrid_clock.cc:648] HybridClock initialized: now 1781874626892292 us; error 56 us; skew 500 ppm
I20260619 13:10:26.892742 15400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:26 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:26.895956 15400 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:26.897392 15400 webserver.cc:460] Webserver started at http://127.0.97.1:38589/ using document root <none> and password file <none>
I20260619 13:10:26.898129 15400 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:26.898263 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:26.904098 15400 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:10:26.909502 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:26.911036 15400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:10:26.911226 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "51ae5600cac2420ca7d5a77bc11abf72"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "c2efed587600c9244088af4d9d40933e"
server_key_iv: "1ea78841296dc5d10f7cb18c154e093b"
server_key_version: "encryptionkey@0"
I20260619 13:10:26.912281 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:26.930131 15400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:26.933966 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:26.934255 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:26.935223 15400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:26.936961 15456 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 13:10:26.942229 15456 ts_tablet_manager.cc:1852] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 13:10:26.943270 15456 ts_tablet_manager.cc:1905] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:26.950927 15456 ts_tablet_manager.cc:1918] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:26.951223 15456 log.cc:1196] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/a9dfcae3c65b4853a16ca26359341c02
I20260619 13:10:26.951746 15456 ts_tablet_manager.cc:1939] T a9dfcae3c65b4853a16ca26359341c02 P 51ae5600cac2420ca7d5a77bc11abf72: Deleting consensus metadata
I20260619 13:10:26.952180 15400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 13:10:26.952319 15400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260619 13:10:26.952476 15400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:26.952564 15400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:26.987280 15400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37879
I20260619 13:10:26.987327 15562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37879 every 8 connection(s)
I20260619 13:10:26.988642 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:26.993438 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:26.991606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:33113)
0619 13:10:26.992321 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:26.992531 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
W20260619 13:10:26.994143 15563 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40361 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40361: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
I20260619 13:10:26.994932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15400
I20260619 13:10:26.995105 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:26.994674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:40299)
0619 13:10:26.994860 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:26.994949 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:10:26.995779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32771
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=34087
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:10:26.996111 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:26.995679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:34255)
0619 13:10:26.995825 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:26.995956 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:10:26.996402 15563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:10:27.326561 15566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:27.326881 15566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:27.326978 15566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:27.334106 15566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:27.334213 15566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:27.334324 15566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:27.343734 15566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34087
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:27.344957 15566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:27.346474 15566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:27.359692 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:27.359625 15572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:27.359792 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:27.360759 15566 server_base.cc:1034] running on GCE node
I20260619 13:10:27.361430 15566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:27.362403 15566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:27.363603 15566 hybrid_clock.cc:648] HybridClock initialized: now 1781874627363568 us; error 43 us; skew 500 ppm
I20260619 13:10:27.363953 15566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:27 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:27.367113 15566 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:27.368472 15566 webserver.cc:460] Webserver started at http://127.0.97.2:34087/ using document root <none> and password file <none>
I20260619 13:10:27.369123 15566 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:27.369225 15566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:27.374697 15566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:10:27.377326 15581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.378746 15566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:10:27.378928 15566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "de7031b9c04749bd823c462adfeb0e61"
format_stamp: "Formatted at 2026-06-19 13:10:21 on dist-test-slave-q11v"
server_key: "12e153ba753ab5160a08413f14f2d7c0"
server_key_iv: "ff13d972a95643b82912bd74897ffd71"
server_key_version: "encryptionkey@0"
I20260619 13:10:27.379914 15566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:27.395489 15566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:27.399940 15566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:27.400239 15566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:27.401225 15566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:27.402670 15566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:27.402810 15566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.402935 15566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:27.403023 15566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.436689 15566 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32771
I20260619 13:10:27.436775 15693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32771 every 8 connection(s)
I20260619 13:10:27.438267 15566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:27.443130 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.441274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:54471)
0619 13:10:27.442063 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.442255 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":352,"threads_started":1}
W20260619 13:10:27.443796 15694 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40361 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40361: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
I20260619 13:10:27.444574 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.444283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:40933)
0619 13:10:27.444412 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.444476 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:10:27.445343 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.445066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:34071)
0619 13:10:27.445178 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.445245 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:10:27.445590 15694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:10:27.448197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15566
I20260619 13:10:27.448716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33795
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40361
--webserver_port=38009
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:27.775345 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:27.775647 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:27.775741 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:27.783257 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:27.783424 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:27.783566 15697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:10:27.793462 15697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38009
--tserver_master_addrs=127.0.97.62:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:27.794817 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:27.796075 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:27.809183 15702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:27.809245 15703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:27.809367 15705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:27.810220 15697 server_base.cc:1034] running on GCE node
I20260619 13:10:27.810796 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:27.811687 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:27.812884 15697 hybrid_clock.cc:648] HybridClock initialized: now 1781874627812816 us; error 55 us; skew 500 ppm
I20260619 13:10:27.813206 15697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:27 dist-test-slave-q11v krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:27.816469 15697 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:27.817961 15697 webserver.cc:460] Webserver started at http://127.0.97.3:38009/ using document root <none> and password file <none>
I20260619 13:10:27.818599 15697 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:27.818707 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:27.823989 15697 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:10:27.826589 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.827898 15697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:10:27.828071 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "27aab526f3fa4befacf7db292a8eb66c"
format_stamp: "Formatted at 2026-06-19 13:10:22 on dist-test-slave-q11v"
server_key: "ba7ad42c7b23fe0a73254740f8cc833e"
server_key_iv: "d28ea185c006c4f1ed9ce8e19061abb3"
server_key_version: "encryptionkey@0"
I20260619 13:10:27.829048 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:27.848905 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:27.853416 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:27.853729 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:27.854722 15697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:27.856052 15697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:27.856142 15697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.856310 15697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:27.856400 15697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:27.889910 15697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33795
I20260619 13:10:27.889964 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33795 every 8 connection(s)
I20260619 13:10:27.891362 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:27.896499 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.894412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:35237)
0619 13:10:27.895297 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.895541 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":388,"threads_started":1}
W20260619 13:10:27.897187 15825 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40361 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40361: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
I20260619 13:10:27.898072 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.897762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:34851)
0619 13:10:27.897895 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.897962 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 13:10:27.898910 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.898562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:48813)
0619 13:10:27.898703 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.898797 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:10:27.899147 15825 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:10:27.901001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15697
I20260619 13:10:27.997839 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:27.996968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:49037)
0619 13:10:27.997429 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:27.997519 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260619 13:10:28.447021 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:28.446205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:55917)
0619 13:10:28.446696 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:28.446789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260619 13:10:28.900688 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:28.899871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:51145)
0619 13:10:28.900342 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:28.900439 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
I20260619 13:10:28.999519 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:28.998797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:33077)
0619 13:10:28.999269 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:28.999357 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260619 13:10:29.448638 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:29.447905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:47869)
0619 13:10:29.448389 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:29.448477 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260619 13:10:29.902411 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:29.901693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:41207)
0619 13:10:29.902150 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:29.902244 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260619 13:10:30.001200 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:30.000455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:49847)
0619 13:10:30.000942 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:30.001029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
I20260619 13:10:30.450183 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:30.449474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:57965)
0619 13:10:30.449935 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:30.450015 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":265,"threads_started":1}
I20260619 13:10:30.903959 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:30.903249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:55233)
0619 13:10:30.903687 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:30.903764 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260619 13:10:31.002936 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:31.002220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:58251)
0619 13:10:31.002697 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:31.002774 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260619 13:10:31.452139 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:31.451321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:37407)
0619 13:10:31.451832 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:31.451927 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
I20260619 13:10:31.905951 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:31.905006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:57617)
0619 13:10:31.905556 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:31.905744 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":283,"threads_started":1}
I20260619 13:10:32.004618 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:32.003872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:37177)
0619 13:10:32.004358 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:32.004445 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260619 13:10:32.454116 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:32.453243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:38125)
0619 13:10:32.453834 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:32.453918 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":285,"threads_started":1}
I20260619 13:10:32.907845 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:32.907070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:36711)
0619 13:10:32.907586 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:32.907667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":294,"threads_started":1}
I20260619 13:10:33.006515 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:33.005673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:43599)
0619 13:10:33.006256 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:33.006343 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
I20260619 13:10:33.456234 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:33.455123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:38433)
0619 13:10:33.455977 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:33.456067 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":532,"threads_started":1}
I20260619 13:10:33.909526 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:33.908774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:51737)
0619 13:10:33.909273 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:33.909363 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
I20260619 13:10:34.008311 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:34.007492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:46829)
0619 13:10:34.008019 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:34.008103 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":231,"threads_started":1}
I20260619 13:10:34.458261 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:34.457291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:41469)
0619 13:10:34.457908 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:34.458013 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":290,"threads_started":1}
I20260619 13:10:34.911276 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:34.910446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:45617)
0619 13:10:34.910962 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:34.911045 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":244,"threads_started":1}
I20260619 13:10:35.010008 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:35.009211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:42349)
0619 13:10:35.009762 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:35.009842 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":255,"threads_started":1}
I20260619 13:10:35.459921 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:35.459218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:44603)
0619 13:10:35.459672 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:35.459755 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260619 13:10:35.913065 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:35.912290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:45189)
0619 13:10:35.912791 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:35.912899 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260619 13:10:36.011840 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:36.010954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:40805)
0619 13:10:36.011558 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:36.011668 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":358,"threads_started":1}
I20260619 13:10:36.461699 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:36.460906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:40185)
0619 13:10:36.461411 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:36.461498 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260619 13:10:36.914815 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:36.914059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:55401)
0619 13:10:36.914539 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:36.914638 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260619 13:10:37.013676 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:37.012913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:46117)
0619 13:10:37.013363 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:37.013442 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260619 13:10:37.463451 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:37.462731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:57299)
0619 13:10:37.463206 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:37.463290 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260619 13:10:37.916810 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:37.915964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:47525)
0619 13:10:37.916508 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:37.916597 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":280,"threads_started":1}
I20260619 13:10:38.015405 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:38.014633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:41173)
0619 13:10:38.015117 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:38.015199 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260619 13:10:38.465217 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:38.464472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:36419)
0619 13:10:38.464938 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:38.465035 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":224,"threads_started":1}
I20260619 13:10:38.918656 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:38.917838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:50133)
0619 13:10:38.918317 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:38.918459 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260619 13:10:39.017144 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:39.016414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:38865)
0619 13:10:39.016886 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:39.016981 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
I20260619 13:10:39.467053 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:39.466378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:41005)
0619 13:10:39.466810 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:39.466891 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260619 13:10:39.920346 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:39.919636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:46845)
0619 13:10:39.920087 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:39.920181 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260619 13:10:40.018916 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:40.018147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:52399)
0619 13:10:40.018620 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:40.018707 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260619 13:10:40.468776 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:40.467995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:40149)
0619 13:10:40.468492 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:40.468580 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":256,"threads_started":1}
I20260619 13:10:40.922204 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:40.921326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:57611)
0619 13:10:40.921925 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:40.922012 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":223,"threads_started":1}
I20260619 13:10:41.020565 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:41.019846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:38551)
0619 13:10:41.020276 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:41.020377 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260619 13:10:41.470789 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:41.470020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:47519)
0619 13:10:41.470513 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:41.470602 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260619 13:10:41.923890 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:41.923075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:49955)
0619 13:10:41.923614 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:41.923702 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":328,"threads_started":1}
I20260619 13:10:42.022578 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:42.021834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.1:58293)
0619 13:10:42.022323 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:42.022405 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":212,"threads_started":1}
I20260619 13:10:42.473018 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:42.472107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.2:37999)
0619 13:10:42.472705 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:42.472811 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
I20260619 13:10:42.909890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15400
I20260619 13:10:42.925954 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:42.925139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40361 (local address 127.0.97.3:45467)
0619 13:10:42.925673 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:42.925770 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260619 13:10:42.929852 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:42.911099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.97.1:37879)
0619 13:10:42.911682 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:42.911697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:42.911737 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:10:42.914053 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:42.914065 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:42.914567 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:42.914971 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:42.916493 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:42.917859 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:42.919120 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:42.919825 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:42.920031 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:42.922903 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:42.922911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:42.922928 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:42.922973 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:42.926458 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:42.927177 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:42.927185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:42.927192 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:42.927357 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:42.927730 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:42.927736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:42.927739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:42.928287 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:42.928608 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:42.929029 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 13:10:42.929463 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":1429,"threads_started":1}
I20260619 13:10:42.931293 15558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:43.017743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15400
I20260619 13:10:43.036124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15566
I20260619 13:10:43.055200 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:43.037066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.97.2:32771)
0619 13:10:43.037681 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:43.037696 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:43.037734 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:10:43.039295 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:43.039305 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:43.039755 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:43.040110 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:43.041562 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:43.042924 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:43.044235 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:43.044948 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:43.045155 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:43.048021 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:43.048029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:43.048046 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:43.048091 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:43.051320 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:43.052213 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:43.052222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:43.052229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:43.052425 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:43.052857 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:43.052864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:43.052868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:43.053470 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:43.053842 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:43.054372 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 13:10:43.054846 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":248,"threads_started":1}
I20260619 13:10:43.056733 15689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:43.141285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15566
I20260619 13:10:43.159073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15697
I20260619 13:10:43.177415 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:43.160054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.0.97.3:33795)
0619 13:10:43.160675 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:43.160690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:43.160732 (+ 42us) server_negotiation.cc:407] Connection header received
0619 13:10:43.162709 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:43.162721 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:43.163197 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:43.163604 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:43.165069 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:43.166385 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:43.167741 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:43.168405 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:43.168612 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:43.171196 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:43.171203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:43.171220 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:43.171257 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:43.174078 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:43.174744 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:43.174751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:43.174757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:43.174920 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:43.175307 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:43.175313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:43.175316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:43.175863 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:43.176171 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:43.176674 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 13:10:43.177107 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
I20260619 13:10:43.178861 15820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:43.263813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15697
2026-06-19T13:10:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24407 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:10:43 dist-test-slave-q11v krb5kdc[15882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:10:43 dist-test-slave-q11v krb5kdc[15882](info): set up 2 sockets
Jun 19 13:10:43 dist-test-slave-q11v krb5kdc[15882](info): commencing operation
krb5kdc: starting...
W20260619 13:10:44.366215 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:10:44 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:10:44.609491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:44.942113 15895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:44.942456 15895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:44.942628 15895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:44.949909 15895 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:10:44.950057 15895 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:10:44.950147 15895 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:10:44.950248 15895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:10:44.950345 15895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:44.950455 15895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:44.950547 15895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:10:44.950639 15895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:10:44.960907 15895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:44.962201 15895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:44.963508 15895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:44.976259 15903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:44.976366 15900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:44.976302 15901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:44.977638 15895 server_base.cc:1034] running on GCE node
I20260619 13:10:44.978772 15895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:44.980360 15895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:44.981727 15895 hybrid_clock.cc:648] HybridClock initialized: now 1781874644981643 us; error 42 us; skew 500 ppm
I20260619 13:10:44.982167 15895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:44 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:44.989267 15895 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:10:44.991163 15895 webserver.cc:460] Webserver started at http://127.0.97.62:42995/ using document root <none> and password file <none>
I20260619 13:10:44.992007 15895 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:44.992154 15895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:44.992512 15895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:44.995940 15895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "965325d4ae084620994beb277dbae21d"
format_stamp: "Formatted at 2026-06-19 13:10:44 on dist-test-slave-q11v"
server_key: "4778f4613ba62a3daae72b07ca8a3922"
server_key_iv: "1b5d0c400da564a7f8bd0fecf7222556"
server_key_version: "encryptionkey@0"
I20260619 13:10:44.996872 15895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "965325d4ae084620994beb277dbae21d"
format_stamp: "Formatted at 2026-06-19 13:10:44 on dist-test-slave-q11v"
server_key: "4778f4613ba62a3daae72b07ca8a3922"
server_key_iv: "1b5d0c400da564a7f8bd0fecf7222556"
server_key_version: "encryptionkey@0"
I20260619 13:10:45.002822 15895 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 13:10:45.006877 15910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:45.008461 15895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:10:45.008714 15895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "965325d4ae084620994beb277dbae21d"
format_stamp: "Formatted at 2026-06-19 13:10:44 on dist-test-slave-q11v"
server_key: "4778f4613ba62a3daae72b07ca8a3922"
server_key_iv: "1b5d0c400da564a7f8bd0fecf7222556"
server_key_version: "encryptionkey@0"
I20260619 13:10:45.008987 15895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:45.032080 15895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:45.037241 15895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:45.037565 15895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:45.058558 15895 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40341
I20260619 13:10:45.058564 15961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40341 every 8 connection(s)
I20260619 13:10:45.060137 15895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:10:45.063081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15895
I20260619 13:10:45.063344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:10:45.064234 388 external_mini_cluster.cc:1428] Setting key 6d52de4b118c001780cd012de0a01308
I20260619 13:10:45.065166 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:10:45 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:45.081039 15962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap starting.
I20260619 13:10:45.085498 15962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:45.086937 15962 log.cc:824] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:45.089340 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:45.066503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.0.97.62:40341)
0619 13:10:45.067295 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:45.067307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:45.067346 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:10:45.068919 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:45.068940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:45.069361 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:45.069857 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:45.071704 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.073123 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.075074 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.075772 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.075943 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:45.080976 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:45.080989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:45.081013 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:45.081072 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:45.085036 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:45.085809 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:45.085820 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:45.085829 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:45.086074 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:45.086446 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:45.086450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:45.086453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:45.087189 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:45.087549 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:45.087885 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 13:10:45.088248 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":375,"threads_started":1}
I20260619 13:10:45.090219 15962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: No bootstrap required, opened a new log
I20260619 13:10:45.098083 15962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:10:45.098551 15962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:45.098678 15962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965325d4ae084620994beb277dbae21d, State: Initialized, Role: FOLLOWER
I20260619 13:10:45.099287 15962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:10:45.099525 15962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:10:45.099679 15962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:10:45.099871 15962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:45.102154 15962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:10:45.102676 15962 leader_election.cc:302] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965325d4ae084620994beb277dbae21d; no voters:
I20260619 13:10:45.103348 15962 leader_election.cc:288] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:10:45.103515 15967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:45.104156 15967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 LEADER]: Becoming Leader. State: Replica: 965325d4ae084620994beb277dbae21d, State: Running, Role: LEADER
I20260619 13:10:45.104718 15967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:10:45.105444 15962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:10:45.108314 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965325d4ae084620994beb277dbae21d. Latest consensus state: current_term: 1 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:10:45.108765 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:45.108366 15968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:10:45.109745 15968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:10:45.110824 15976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:10:45.116104 15976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:10:45.124554 15976 catalog_manager.cc:1347] Generated new cluster ID: 3e57a68d67a64efa94f4b2a82c8b5efb
I20260619 13:10:45.124682 15976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:10:45.140548 15976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:10:45.142580 15976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:10:45.152288 15976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Generated new TSK 0
I20260619 13:10:45.153918 15976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:10:45.215658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:45.550973 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:45.551250 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:45.551411 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:45.558804 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:45.558920 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:45.559118 15988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:10:45.568975 15988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:45.570349 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:45.571679 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:45.585031 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:45.585268 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:45.585539 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:45.585911 15988 server_base.cc:1034] running on GCE node
I20260619 13:10:45.586531 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:45.587509 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:45.588735 15988 hybrid_clock.cc:648] HybridClock initialized: now 1781874645588670 us; error 65 us; skew 500 ppm
I20260619 13:10:45.589067 15988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:45 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:45.592304 15988 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:10:45.593803 15988 webserver.cc:460] Webserver started at http://127.0.97.1:38117/ using document root <none> and password file <none>
I20260619 13:10:45.594502 15988 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:45.594655 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:45.594974 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:45.598037 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
format_stamp: "Formatted at 2026-06-19 13:10:45 on dist-test-slave-q11v"
server_key: "aef74954e13da16eec6bc89fc1de1625"
server_key_iv: "99901fcf7638aaf86a01a5c3b404dce1"
server_key_version: "encryptionkey@0"
I20260619 13:10:45.598819 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
format_stamp: "Formatted at 2026-06-19 13:10:45 on dist-test-slave-q11v"
server_key: "aef74954e13da16eec6bc89fc1de1625"
server_key_iv: "99901fcf7638aaf86a01a5c3b404dce1"
server_key_version: "encryptionkey@0"
I20260619 13:10:45.604141 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 13:10:45.607734 16003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:45.609140 15988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 13:10:45.609340 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
format_stamp: "Formatted at 2026-06-19 13:10:45 on dist-test-slave-q11v"
server_key: "aef74954e13da16eec6bc89fc1de1625"
server_key_iv: "99901fcf7638aaf86a01a5c3b404dce1"
server_key_version: "encryptionkey@0"
I20260619 13:10:45.609577 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:45.628329 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:45.632486 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:45.632740 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:45.633726 15988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:45.635100 15988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:45.635177 15988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:45.635267 15988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:45.635311 15988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:45.669466 15988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45167
I20260619 13:10:45.669513 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45167 every 8 connection(s)
I20260619 13:10:45.670785 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:10:45.672087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 15988
I20260619 13:10:45.672308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:10:45.673168 388 external_mini_cluster.cc:1428] Setting key 84dd637ecb178b44c641e2b5ebf43c0f
Jun 19 13:10:45 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:45.691540 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:45.673791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54581 (local address 127.0.97.62:40341)
0619 13:10:45.674275 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:45.674281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:45.675285 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 13:10:45.677323 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:45.677329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:45.677429 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:45.677554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:45.679131 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.679857 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.681460 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.681918 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.682062 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:45.685660 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:45.685666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:45.685670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:45.685703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:45.688319 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:45.689087 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:45.689095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:45.689099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:45.689223 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:45.689724 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:45.689731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:45.689734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:45.690015 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:45.690177 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:45.690918 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 13:10:45.691156 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
I20260619 13:10:45.692559 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:45.674444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:54581)
0619 13:10:45.675135 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:45.675176 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:45.677051 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:45.677831 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:45.677840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:45.678381 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:45.678958 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:45.678976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.680037 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.680041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:45.681298 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:45.681310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:45.682163 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:45.682171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:45.682421 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:45.683122 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:45.683145 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:45.685367 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:45.688474 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:45.688484 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:45.688503 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:45.688963 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:45.689331 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:45.689336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:45.689339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:45.689562 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:45.690149 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:45.690157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:45.690532 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 13:10:45.690814 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 13:10:45.691393 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":749,"threads_started":1}
I20260619 13:10:45.694259 16116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:10:45.694576 16116 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:10:45.695344 16116 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:10:45.697702 15927 ts_manager.cc:194] Registered new tserver with Master: ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167)
I20260619 13:10:45.699195 15927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:10:45.726276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:46.055874 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:46.056186 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:46.056354 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:46.064013 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:46.064132 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:46.064309 16122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:10:46.074476 16122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:46.075810 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:46.077245 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:46.090874 16128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:46.091012 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:46.090987 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:46.091931 16122 server_base.cc:1034] running on GCE node
I20260619 13:10:46.092545 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:46.093573 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:46.094868 16122 hybrid_clock.cc:648] HybridClock initialized: now 1781874646094813 us; error 41 us; skew 500 ppm
I20260619 13:10:46.095216 16122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:46 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:46.098562 16122 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:10:46.100114 16122 webserver.cc:460] Webserver started at http://127.0.97.2:33329/ using document root <none> and password file <none>
I20260619 13:10:46.100849 16122 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:46.101006 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:46.101334 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:46.104573 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "f695a0054ed44edb8e7dfcb6aac63713"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "ee20942e41dd2abeb1c2693b13abd8dc"
server_key_iv: "3110c2e25ff1d962ed4fcdadbe3fbc3e"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.105494 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f695a0054ed44edb8e7dfcb6aac63713"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "ee20942e41dd2abeb1c2693b13abd8dc"
server_key_iv: "3110c2e25ff1d962ed4fcdadbe3fbc3e"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.111363 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:10:46.115456 16137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.116928 16122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:10:46.117141 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "f695a0054ed44edb8e7dfcb6aac63713"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "ee20942e41dd2abeb1c2693b13abd8dc"
server_key_iv: "3110c2e25ff1d962ed4fcdadbe3fbc3e"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.117389 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:46.136760 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:46.141405 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:46.141845 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:46.145120 16122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:46.147141 16122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:46.147261 16122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.147395 16122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:46.147477 16122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.181566 16122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40069
I20260619 13:10:46.181644 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40069 every 8 connection(s)
I20260619 13:10:46.182955 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:10:46.190413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16122
I20260619 13:10:46.190630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:10:46.191109 388 external_mini_cluster.cc:1428] Setting key c40abe046bf700949be843113981f2f6
Jun 19 13:10:46 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:46.203485 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.185556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55237 (local address 127.0.97.62:40341)
0619 13:10:46.185765 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.185770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.186967 (+ 1197us) server_negotiation.cc:407] Connection header received
0619 13:10:46.188837 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.188843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.188941 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.189073 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:46.190735 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.191516 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.193405 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.193910 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.194058 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.197562 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:46.197568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:46.197571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:46.197650 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:46.200316 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.201069 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.201077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.201081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.201216 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.201707 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.201714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.201717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.202002 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:46.202172 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.202886 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.203138 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 13:10:46.204501 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.186083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:55237)
0619 13:10:46.186800 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:46.186843 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:46.188550 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:46.189317 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:46.189327 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:46.189932 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:46.190547 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.190579 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.191687 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.191692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.193256 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.193267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.194145 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.194152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.194385 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.195130 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:46.195159 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:46.197316 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:46.200472 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:46.200481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:46.200500 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:46.200933 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:46.201335 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:46.201340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:46.201344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:46.201542 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:46.202143 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:46.202151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:46.202506 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 13:10:46.202790 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 13:10:46.203337 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":282,"threads_started":1}
I20260619 13:10:46.206091 16250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:10:46.206440 16250 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:46.207378 16250 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
I20260619 13:10:46.209081 15927 ts_manager.cc:194] Registered new tserver with Master: f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069)
I20260619 13:10:46.210436 15927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55237
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:10:46.245070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:10:46.576628 16255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:10:46.576952 16255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:10:46.577047 16255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:10:46.584398 16255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:10:46.584523 16255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:10:46.584643 16255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:10:46.594689 16255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:10:46.596051 16255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:10:46.597404 16255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:10:46.611049 16263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:46.611107 16261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:10:46.611013 16260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:10:46.612625 16255 server_base.cc:1034] running on GCE node
I20260619 13:10:46.613411 16255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:10:46.614584 16255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:10:46.615831 16255 hybrid_clock.cc:648] HybridClock initialized: now 1781874646615768 us; error 56 us; skew 500 ppm
I20260619 13:10:46.616176 16255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:10:46 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:10:46.619406 16255 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:10:46.620862 16255 webserver.cc:460] Webserver started at http://127.0.97.3:36461/ using document root <none> and password file <none>
I20260619 13:10:46.621560 16255 fs_manager.cc:359] Metadata directory not provided
I20260619 13:10:46.621754 16255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:10:46.622052 16255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:10:46.624992 16255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "dcecc7807646a208374cde90c1644566"
server_key_iv: "67328397abd4ec8ac49a15086fc97aa7"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.625795 16255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "dcecc7807646a208374cde90c1644566"
server_key_iv: "67328397abd4ec8ac49a15086fc97aa7"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.630833 16255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:10:46.634272 16270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.635600 16255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:10:46.635769 16255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "dcecc7807646a208374cde90c1644566"
server_key_iv: "67328397abd4ec8ac49a15086fc97aa7"
server_key_version: "encryptionkey@0"
I20260619 13:10:46.635979 16255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:10:46.649789 16255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:10:46.653524 16255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:10:46.653841 16255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:10:46.654771 16255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:10:46.656050 16255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:10:46.656127 16255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.656302 16255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:10:46.656380 16255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:10:46.690268 16255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37241
I20260619 13:10:46.690304 16382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37241 every 8 connection(s)
I20260619 13:10:46.691593 16255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:10:46.698082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16255
I20260619 13:10:46.698289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:10:46.698868 388 external_mini_cluster.cc:1428] Setting key f6c6edaa5c6c88221d66f4baeb4e6f4c
I20260619 13:10:46.702174 16116 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
Jun 19 13:10:46 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:10:46.712373 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.694212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34379 (local address 127.0.97.62:40341)
0619 13:10:46.694390 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.694394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.695575 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 13:10:46.697360 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.697364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.697433 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.697524 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:46.699658 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.700706 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.702512 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.703052 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.703192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.706839 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:46.706844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:46.706847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:46.706879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:46.709515 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.710220 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.710224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.710226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.710370 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.710759 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.710762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.710764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.711099 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:46.711271 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.711956 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.712129 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:10:46.713899 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.694695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:34379)
0619 13:10:46.695406 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:46.695443 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:46.697103 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:46.697857 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:46.697873 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:46.698658 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:10:46.699472 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.699492 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.700882 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.700886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.702366 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.702379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.703278 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.703286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.703543 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.704311 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:10:46.704334 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:10:46.706636 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:10:46.709712 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:46.709718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:46.709730 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:46.710129 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:46.710461 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:10:46.710464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:10:46.710466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:10:46.710656 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:10:46.711252 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:10:46.711258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:10:46.711620 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 13:10:46.711837 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 13:10:46.712268 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":294,"threads_started":1}
I20260619 13:10:46.715878 16383 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:10:46.716238 16383 heartbeater.cc:464] Registering TS with master...
I20260619 13:10:46.716917 16383 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
I20260619 13:10:46.718268 15927 ts_manager.cc:194] Registered new tserver with Master: ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
I20260619 13:10:46.719203 15927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34379
I20260619 13:10:46.728387 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:10:46.745847 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.732899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.0.97.62:40341)
0619 13:10:46.733069 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.733073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.733536 (+ 463us) server_negotiation.cc:407] Connection header received
0619 13:10:46.733761 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.733764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.733839 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.733924 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:46.735231 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.735942 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.737300 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.737823 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.737944 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.740257 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:46.740263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:46.740266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:46.740298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:46.742788 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.743501 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.743506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.743511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.743639 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:46.744237 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:46.744245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:46.744249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:46.744601 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:46.744898 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.745241 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.745490 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:10:46.749969 15927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:10:46.756794 15927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:10:46.790650 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.768968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37241 (local address 127.0.0.1:52720)
0619 13:10:46.782356 (+ 13388us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:46.782375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:46.782639 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:46.783655 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:46.783662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:46.783686 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:46.784128 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.784141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.786006 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.786012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.788128 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.788140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.789413 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.789419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.789674 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.789739 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 13:10:46.789885 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 13:10:46.790089 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13247,"mutex_wait_us":146,"spinlock_wait_cycles":7936,"thread_start_us":12829,"threads_started":1}
I20260619 13:10:46.790681 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.769024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.97.1:45167)
0619 13:10:46.782162 (+ 13138us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.782169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.782196 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:10:46.782318 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.782327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.782660 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.782878 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:46.783674 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.784947 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.787964 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.789469 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.789640 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.789869 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.790157 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.790340 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12988,"thread_start_us":12815,"threads_started":1}
I20260619 13:10:46.791044 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.769689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.0.97.3:37241)
0619 13:10:46.770638 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.770652 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.782493 (+ 11841us) server_negotiation.cc:407] Connection header received
0619 13:10:46.782880 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.782891 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.783163 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.783338 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:46.784309 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.785870 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.788283 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.789213 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.789357 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.789604 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.790515 (+ 911us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.790696 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":496,"threads_started":1}
I20260619 13:10:46.791594 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.768678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:33974)
0619 13:10:46.780382 (+ 11704us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:46.780433 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:46.780638 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:46.782960 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:46.782968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:46.782998 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:46.783446 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.783462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.785083 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.785091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.787778 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.787793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.789784 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.789790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.789965 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.789994 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:10:46.790072 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:10:46.790190 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11441,"mutex_wait_us":148,"thread_start_us":12746,"threads_started":1}
I20260619 13:10:46.791199 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.772297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40069 (local address 127.0.0.1:47568)
0619 13:10:46.782451 (+ 10154us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:46.782470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:46.782625 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:46.783385 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:46.783393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:46.783421 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:46.783841 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.783851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.785724 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.785730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.788165 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:46.788178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.790264 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.790272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:46.790453 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.790488 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:10:46.790581 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 13:10:46.790743 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10010,"mutex_wait_us":157,"spinlock_wait_cycles":4864,"thread_start_us":7677,"threads_started":1}
I20260619 13:10:46.792479 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:46.773397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.97.2:40069)
0619 13:10:46.779778 (+ 6381us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:46.779792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:46.782811 (+ 3019us) server_negotiation.cc:407] Connection header received
0619 13:10:46.782930 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:46.782938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:46.783188 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:46.783400 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:46.784021 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.785581 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.788341 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:46.789796 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:46.789924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:46.790101 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:46.791321 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0619 13:10:46.791486 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6162,"thread_start_us":6010,"threads_started":1}
I20260619 13:10:46.795547 16318 tablet_service.cc:1467] Processing CreateTablet for tablet bab05f976371482e9a4c8027dd91207a (DEFAULT_TABLE table=test-table [id=7937525843bd4ddfbacaf1b53272e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:46.795547 16051 tablet_service.cc:1467] Processing CreateTablet for tablet bab05f976371482e9a4c8027dd91207a (DEFAULT_TABLE table=test-table [id=7937525843bd4ddfbacaf1b53272e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:46.795799 16185 tablet_service.cc:1467] Processing CreateTablet for tablet bab05f976371482e9a4c8027dd91207a (DEFAULT_TABLE table=test-table [id=7937525843bd4ddfbacaf1b53272e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:10:46.796976 16185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab05f976371482e9a4c8027dd91207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:46.797111 16051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab05f976371482e9a4c8027dd91207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:46.797128 16318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab05f976371482e9a4c8027dd91207a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:10:46.810781 16401 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Bootstrap starting.
I20260619 13:10:46.811661 16403 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Bootstrap starting.
I20260619 13:10:46.814409 16402 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Bootstrap starting.
I20260619 13:10:46.814970 16401 tablet_bootstrap.cc:654] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:46.815937 16403 tablet_bootstrap.cc:654] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:46.816310 16401 log.cc:824] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:46.817214 16403 log.cc:824] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:46.817525 16402 tablet_bootstrap.cc:654] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Neither blocks nor log segments found. Creating new log.
I20260619 13:10:46.818502 16402 log.cc:824] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Log is configured to *not* fsync() on all Append() calls
I20260619 13:10:46.819977 16401 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: No bootstrap required, opened a new log
I20260619 13:10:46.821007 16401 ts_tablet_manager.cc:1397] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:10:46.821038 16403 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: No bootstrap required, opened a new log
I20260619 13:10:46.821293 16403 ts_tablet_manager.cc:1397] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 13:10:46.821699 16402 tablet_bootstrap.cc:492] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: No bootstrap required, opened a new log
I20260619 13:10:46.821952 16402 ts_tablet_manager.cc:1397] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260619 13:10:46.828558 16403 raft_consensus.cc:348] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.828990 16403 raft_consensus.cc:374] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:46.829092 16403 raft_consensus.cc:729] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceaa825ea4f3431e96b0d1a52db549f0, State: Initialized, Role: FOLLOWER
I20260619 13:10:46.829855 16403 consensus_queue.cc:260] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.832538 16402 raft_consensus.cc:348] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.832875 16403 ts_tablet_manager.cc:1428] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260619 13:10:46.832962 16402 raft_consensus.cc:374] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:46.833068 16402 raft_consensus.cc:729] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebeab24bbcf24e7cb6e40d7a071f8e84, State: Initialized, Role: FOLLOWER
I20260619 13:10:46.832216 16401 raft_consensus.cc:348] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.833813 16401 raft_consensus.cc:374] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:10:46.833947 16401 raft_consensus.cc:729] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f695a0054ed44edb8e7dfcb6aac63713, State: Initialized, Role: FOLLOWER
I20260619 13:10:46.833827 16402 consensus_queue.cc:260] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.834530 16401 consensus_queue.cc:260] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:46.835637 16250 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
I20260619 13:10:46.836284 16401 ts_tablet_manager.cc:1428] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 13:10:46.838128 16383 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
I20260619 13:10:46.838769 16402 ts_tablet_manager.cc:1428] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260619 13:10:46.923444 16117 tablet.cc:2369] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:10:46.935050 16251 tablet.cc:2369] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:10:46.943322 16384 tablet.cc:2369] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:10:47.147974 16408 raft_consensus.cc:482] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:10:47.148298 16408 raft_consensus.cc:504] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:47.150058 16408 leader_election.cc:288] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167), f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069)
I20260619 13:10:47.156562 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.150350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40069 (local address 127.0.97.3:42399)
0619 13:10:47.150675 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:47.150704 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:47.150873 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:47.151326 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:47.151330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:47.151439 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:47.151645 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:47.151659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.152877 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.152883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:47.154543 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:47.154554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.155786 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.155792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:47.155979 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.156029 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:10:47.156121 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 13:10:47.156277 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":139,"spinlock_wait_cycles":25984}
I20260619 13:10:47.156600 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.150559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42399 (local address 127.0.97.2:40069)
0619 13:10:47.150767 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.150772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.150798 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:10:47.151069 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.151075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.151159 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.151280 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:47.151795 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.152744 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.154700 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.155614 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.155720 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.155794 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.156172 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.156308 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:10:47.156948 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.150649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42459 (local address 127.0.97.1:45167)
0619 13:10:47.150839 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.150844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.151347 (+ 503us) server_negotiation.cc:407] Connection header received
0619 13:10:47.151507 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.151512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.151589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.151691 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:10:47.152317 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.153148 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.155041 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.155946 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.156089 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.156204 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.156536 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.156684 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:10:47.157033 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.150409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.97.3:42459)
0619 13:10:47.151224 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 13:10:47.151243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:10:47.151367 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:10:47.151729 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:10:47.151736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:10:47.151762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:10:47.152145 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:47.152159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.153295 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.153299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:47.154881 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:10:47.154894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.156182 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.156187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:10:47.156355 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.156393 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:10:47.156487 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 13:10:47.156614 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":21,"thread_start_us":538,"threads_started":1}
I20260619 13:10:47.157752 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab05f976371482e9a4c8027dd91207a" candidate_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f695a0054ed44edb8e7dfcb6aac63713" is_pre_election: true
I20260619 13:10:47.158119 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab05f976371482e9a4c8027dd91207a" candidate_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" is_pre_election: true
I20260619 13:10:47.158270 16205 raft_consensus.cc:2454] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebeab24bbcf24e7cb6e40d7a071f8e84 in term 0.
I20260619 13:10:47.158614 16071 raft_consensus.cc:2454] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebeab24bbcf24e7cb6e40d7a071f8e84 in term 0.
I20260619 13:10:47.158926 16272 leader_election.cc:302] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713; no voters:
I20260619 13:10:47.159384 16408 raft_consensus.cc:2781] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:10:47.159538 16408 raft_consensus.cc:482] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:10:47.159657 16408 raft_consensus.cc:3037] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:47.161970 16408 raft_consensus.cc:504] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:47.162628 16408 leader_election.cc:288] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [CANDIDATE]: Term 1 election: Requested vote from peers ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167), f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069)
I20260619 13:10:47.163198 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab05f976371482e9a4c8027dd91207a" candidate_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f695a0054ed44edb8e7dfcb6aac63713"
I20260619 13:10:47.163205 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab05f976371482e9a4c8027dd91207a" candidate_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
I20260619 13:10:47.163419 16205 raft_consensus.cc:3037] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:47.163445 16071 raft_consensus.cc:3037] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:10:47.167057 16071 raft_consensus.cc:2454] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebeab24bbcf24e7cb6e40d7a071f8e84 in term 1.
I20260619 13:10:47.167063 16205 raft_consensus.cc:2454] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebeab24bbcf24e7cb6e40d7a071f8e84 in term 1.
I20260619 13:10:47.167522 16272 leader_election.cc:302] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713; no voters:
I20260619 13:10:47.167860 16408 raft_consensus.cc:2781] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:10:47.168368 16408 raft_consensus.cc:686] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 LEADER]: Becoming Leader. State: Replica: ebeab24bbcf24e7cb6e40d7a071f8e84, State: Running, Role: LEADER
I20260619 13:10:47.169006 16408 consensus_queue.cc:237] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } }
I20260619 13:10:47.174811 15927 catalog_manager.cc:5495] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } health_report { overall_health: UNKNOWN } } }
I20260619 13:10:47.241453 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.235108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.0.97.3:37241)
0619 13:10:47.235346 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.235354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.235472 (+ 118us) server_negotiation.cc:407] Connection header received
0619 13:10:47.235771 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.235776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.235865 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.236113 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:10:47.236497 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.237314 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.239085 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.239546 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.239698 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.240252 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:10:47.240812 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:10:47.240996 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.241063 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.241154 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:10:47.250098 16071 raft_consensus.cc:1261] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Refusing update from remote peer ebeab24bbcf24e7cb6e40d7a071f8e84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:47.250109 16205 raft_consensus.cc:1261] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Refusing update from remote peer ebeab24bbcf24e7cb6e40d7a071f8e84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:10:47.251204 16408 consensus_queue.cc:1035] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:47.251519 16411 consensus_queue.cc:1035] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:10:47.278180 15927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:47.278481 15927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:10:47.282150 15927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Sending DeleteTablet for 3 replicas of tablet bab05f976371482e9a4c8027dd91207a
I20260619 13:10:47.283366 16318 tablet_service.cc:1514] Processing DeleteTablet for tablet bab05f976371482e9a4c8027dd91207a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52720
I20260619 13:10:47.283375 16051 tablet_service.cc:1514] Processing DeleteTablet for tablet bab05f976371482e9a4c8027dd91207a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33974
I20260619 13:10:47.284099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15988
I20260619 13:10:47.284137 16185 tablet_service.cc:1514] Processing DeleteTablet for tablet bab05f976371482e9a4c8027dd91207a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47568
I20260619 13:10:47.294976 16424 tablet_replica.cc:330] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: stopping tablet replica
I20260619 13:10:47.295490 16424 raft_consensus.cc:2229] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:47.296103 16424 raft_consensus.cc:2258] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:47.297039 16423 tablet_replica.cc:330] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: stopping tablet replica
I20260619 13:10:47.297501 16423 raft_consensus.cc:2229] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:10:47.298228 16423 raft_consensus.cc:2258] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:47.300688 16424 ts_tablet_manager.cc:1905] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:47.300686 16423 ts_tablet_manager.cc:1905] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:10:47 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:10:47.308400 16424 ts_tablet_manager.cc:1918] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:47.308581 16424 log.cc:1196] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/bab05f976371482e9a4c8027dd91207a
I20260619 13:10:47.309152 16424 ts_tablet_manager.cc:1939] T bab05f976371482e9a4c8027dd91207a P ceaa825ea4f3431e96b0d1a52db549f0: Deleting consensus metadata
I20260619 13:10:47.309320 16425 tablet_replica.cc:330] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: stopping tablet replica
I20260619 13:10:47.309870 16425 raft_consensus.cc:2229] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:10:47.310412 16425 raft_consensus.cc:2258] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:10:47.311597 15913 catalog_manager.cc:4841] TS ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167): tablet bab05f976371482e9a4c8027dd91207a (table test-table [id=7937525843bd4ddfbacaf1b53272e51b]) successfully deleted
I20260619 13:10:47.313900 16423 ts_tablet_manager.cc:1918] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:47.314157 16423 log.cc:1196] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/bab05f976371482e9a4c8027dd91207a
I20260619 13:10:47.314584 16425 ts_tablet_manager.cc:1905] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:10:47.314743 16423 ts_tablet_manager.cc:1939] T bab05f976371482e9a4c8027dd91207a P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting consensus metadata
I20260619 13:10:47.316526 15914 catalog_manager.cc:4841] TS ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): tablet bab05f976371482e9a4c8027dd91207a (table test-table [id=7937525843bd4ddfbacaf1b53272e51b]) successfully deleted
I20260619 13:10:47.316917 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.285493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.97.1:45167)
0619 13:10:47.285754 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.285764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.294706 (+ 8942us) server_negotiation.cc:407] Connection header received
0619 13:10:47.295113 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.295119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.295211 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.295288 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:47.297895 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.299049 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.301551 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.302227 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.302331 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.308137 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:47.308162 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:47.308182 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:47.308231 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:47.313296 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.314207 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.314216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.314220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.314425 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.314915 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.314920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.314922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.315381 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:47.315688 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.316282 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.316513 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:10:47.318257 16111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:47.324410 16425 ts_tablet_manager.cc:1918] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:10:47.324651 16425 log.cc:1196] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/bab05f976371482e9a4c8027dd91207a
I20260619 13:10:47.325330 16425 ts_tablet_manager.cc:1939] T bab05f976371482e9a4c8027dd91207a P f695a0054ed44edb8e7dfcb6aac63713: Deleting consensus metadata
I20260619 13:10:47.327648 15912 catalog_manager.cc:4841] TS f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): tablet bab05f976371482e9a4c8027dd91207a (table test-table [id=7937525843bd4ddfbacaf1b53272e51b]) successfully deleted
I20260619 13:10:47.437743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15988
I20260619 13:10:47.456869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16122
Jun 19 13:10:47 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:10:47.474992 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.457956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.0.97.2:40069)
0619 13:10:47.458141 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.458148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.458171 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:10:47.458400 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.458405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.458479 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.458604 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:47.460050 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.460775 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.462391 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.462795 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.462937 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.468125 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:47.468147 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:47.468169 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:47.468219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:47.471916 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.472615 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.472625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.472629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.472786 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.473126 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.473133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.473138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.473853 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:47.474151 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.474369 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.474606 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:10:47.475669 16245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:47.566797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16122
I20260619 13:10:47.590817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16255
Jun 19 13:10:47 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:10:47.607470 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:10:47.591641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.97.3:37241)
0619 13:10:47.591837 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:10:47.591843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:10:47.591922 (+ 79us) server_negotiation.cc:407] Connection header received
0619 13:10:47.592136 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:10:47.592140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:10:47.592215 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:10:47.592323 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:10:47.593646 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.594497 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.595952 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:10:47.596422 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:10:47.596560 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:10:47.601264 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:10:47.601277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:10:47.601293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:10:47.601328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:10:47.604497 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.605218 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.605226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.605228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.605382 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:10:47.605871 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:10:47.605877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:10:47.605881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:10:47.606265 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:10:47.606651 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0619 13:10:47.606844 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 13:10:47.607079 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:10:47.608203 16378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:47.731598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16255
I20260619 13:10:47.751176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 15895
I20260619 13:10:47.752377 15957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:10:47.866720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 15895
I20260619 13:11:07.888419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--webserver_interface=127.0.97.62
--webserver_port=42995
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:08.223834 16431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:08.224217 16431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:08.224382 16431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:08.232496 16431 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:11:08.232635 16431 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:11:08.232705 16431 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:11:08.232780 16431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:11:08.232860 16431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:08.232970 16431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:08.233062 16431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:11:08.233141 16431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:11:08.243392 16431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:08.244809 16431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:08.246352 16431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:08.270081 16440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:08.270061 16438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:08.270779 16437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:08.270774 16431 server_base.cc:1034] running on GCE node
I20260619 13:11:08.271812 16431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:08.273464 16431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:08.274716 16431 hybrid_clock.cc:648] HybridClock initialized: now 1781874668274633 us; error 59 us; skew 500 ppm
I20260619 13:11:08.275138 16431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:08 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:08.283535 16431 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:11:08.285326 16431 webserver.cc:460] Webserver started at http://127.0.97.62:42995/ using document root <none> and password file <none>
I20260619 13:11:08.286199 16431 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:08.286307 16431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:08.292480 16431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:11:08.295552 16447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:08.297024 16431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:11:08.297230 16431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "965325d4ae084620994beb277dbae21d"
format_stamp: "Formatted at 2026-06-19 13:10:44 on dist-test-slave-q11v"
server_key: "4778f4613ba62a3daae72b07ca8a3922"
server_key_iv: "1b5d0c400da564a7f8bd0fecf7222556"
server_key_version: "encryptionkey@0"
I20260619 13:11:08.298373 16431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:08.321864 16431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:08.326756 16431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:08.327085 16431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:08.346765 16431 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40341
I20260619 13:11:08.346788 16498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40341 every 8 connection(s)
I20260619 13:11:08.348407 16431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:11:08.352494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16431
I20260619 13:11:08.353360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45167
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=38117
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:11:08.355773 16499 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:11:08.359265 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap starting.
I20260619 13:11:08.379522 16499 log.cc:824] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:08.391611 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:11:08.392232 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap complete.
I20260619 13:11:08.401487 16499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:08.402520 16499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965325d4ae084620994beb277dbae21d, State: Initialized, Role: FOLLOWER
I20260619 13:11:08.403196 16499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:08.403501 16499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:11:08.403645 16499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:11:08.403859 16499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:11:08.407770 16499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:08.408310 16499 leader_election.cc:302] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965325d4ae084620994beb277dbae21d; no voters:
I20260619 13:11:08.408980 16499 leader_election.cc:288] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:11:08.409118 16503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:11:08.410475 16503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 LEADER]: Becoming Leader. State: Replica: 965325d4ae084620994beb277dbae21d, State: Running, Role: LEADER
I20260619 13:11:08.411031 16503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:08.411477 16499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:11:08.413789 16505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965325d4ae084620994beb277dbae21d. Latest consensus state: current_term: 2 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:11:08.414187 16505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:08.413425 16504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:11:08.414765 16504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:08.419597 16509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:11:08.429139 16509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:11:08.430721 16509 catalog_manager.cc:1259] Loaded cluster ID: 3e57a68d67a64efa94f4b2a82c8b5efb
I20260619 13:11:08.430877 16509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:11:08.435019 16509 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:11:08.436578 16509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Loaded TSK: 0
I20260619 13:11:08.467388 16509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Generated new TSK 1
I20260619 13:11:08.467832 16509 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:11:08.709468 16501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:08.709882 16501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:08.710044 16501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:08.717948 16501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:08.718132 16501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:08.718324 16501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:11:08.729024 16501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38117
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:08.730535 16501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:08.731997 16501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:08.751642 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:08.751816 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:08.752830 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:08.754313 16501 server_base.cc:1034] running on GCE node
I20260619 13:11:08.755342 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:08.756884 16501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:08.758137 16501 hybrid_clock.cc:648] HybridClock initialized: now 1781874668758057 us; error 62 us; skew 500 ppm
I20260619 13:11:08.758603 16501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:08 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:08.762681 16501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:11:08.767230 16501 webserver.cc:460] Webserver started at http://127.0.97.1:38117/ using document root <none> and password file <none>
I20260619 13:11:08.768038 16501 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:08.768196 16501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:08.775046 16501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:11:08.778654 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:08.780470 16501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:11:08.780670 16501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
format_stamp: "Formatted at 2026-06-19 13:10:45 on dist-test-slave-q11v"
server_key: "aef74954e13da16eec6bc89fc1de1625"
server_key_iv: "99901fcf7638aaf86a01a5c3b404dce1"
server_key_version: "encryptionkey@0"
I20260619 13:11:08.782328 16501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:08.807241 16501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:08.812067 16501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:08.812353 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:08.813568 16501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:08.815138 16501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:08.815256 16501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:08.815388 16501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:08.815479 16501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:08.856469 16501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45167
I20260619 13:11:08.856530 16648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45167 every 8 connection(s)
I20260619 13:11:08.858055 16501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:11:08.868474 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16501
I20260619 13:11:08.869122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40069
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=33329
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:11:08 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:08.883616 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:08.861459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:43025)
0619 13:11:08.862339 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:08.862386 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:08.864101 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:08.865540 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:08.865553 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:08.866242 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:08.866796 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:08.866814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:08.868535 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:08.868539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:08.870014 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:08.870027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:08.871100 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:08.871108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:08.871375 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:08.872076 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:08.872107 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:08.874705 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:08.879432 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:08.879441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:08.879456 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:08.879901 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:08.880417 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:08.880421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:08.880423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:08.880616 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:08.881570 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:08.881577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:08.881935 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 13:11:08.882175 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 13:11:08.882593 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":348,"threads_started":1}
I20260619 13:11:08.884186 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:08.861232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43025 (local address 127.0.97.62:40341)
0619 13:11:08.861996 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:08.862016 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:08.862521 (+ 505us) server_negotiation.cc:407] Connection header received
0619 13:11:08.864659 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:08.864671 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:08.864998 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:08.865394 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:08.867034 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:08.868348 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:08.870167 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:08.870843 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:08.871057 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:08.875023 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:08.875031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:08.875048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:08.875097 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:08.879286 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:08.880056 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:08.880067 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:08.880079 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:08.880302 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:08.880730 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:08.880737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:08.880744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:08.881421 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:08.881659 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:08.882443 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 13:11:08.882891 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":352,"threads_started":1}
I20260619 13:11:08.887125 16649 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:11:08.887454 16649 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:08.888458 16649 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
I20260619 13:11:08.890991 16464 ts_manager.cc:194] Registered new tserver with Master: ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167)
I20260619 13:11:08.892488 16464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43025
W20260619 13:11:09.199446 16653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:09.199805 16653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:09.199980 16653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:09.207515 16653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:09.207700 16653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:09.207878 16653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:11:09.218129 16653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33329
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:09.219527 16653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:09.220988 16653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:09.234694 16658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:09.234779 16659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:09.234793 16661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:09.239200 16653 server_base.cc:1034] running on GCE node
I20260619 13:11:09.239848 16653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:09.240895 16653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:09.242101 16653 hybrid_clock.cc:648] HybridClock initialized: now 1781874669242049 us; error 49 us; skew 500 ppm
I20260619 13:11:09.242455 16653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:09 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:09.245806 16653 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:11:09.247416 16653 webserver.cc:460] Webserver started at http://127.0.97.2:33329/ using document root <none> and password file <none>
I20260619 13:11:09.248116 16653 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:09.248229 16653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:09.253810 16653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:11:09.256699 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.258189 16653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:11:09.258358 16653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "f695a0054ed44edb8e7dfcb6aac63713"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "ee20942e41dd2abeb1c2693b13abd8dc"
server_key_iv: "3110c2e25ff1d962ed4fcdadbe3fbc3e"
server_key_version: "encryptionkey@0"
I20260619 13:11:09.259349 16653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:09.276355 16653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:09.280678 16653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:09.280959 16653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:09.282003 16653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:09.283336 16653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:09.283447 16653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.283578 16653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:09.283672 16653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.317412 16653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40069
I20260619 13:11:09.317508 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40069 every 8 connection(s)
I20260619 13:11:09.318997 16653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:11:09.326058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16653
I20260619 13:11:09.326699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37241
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=36461
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:11:09 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:09.341212 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.321645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41137 (local address 127.0.97.62:40341)
0619 13:11:09.321854 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.321861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.323091 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 13:11:09.324943 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.324947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.325021 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.325113 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:09.327021 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.327817 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.329472 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.330187 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.330318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.335024 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:09.335029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:09.335032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:09.335075 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:09.337828 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.338822 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.338826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.338828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.338947 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.339450 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.339453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.339455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.339725 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:09.339834 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.340784 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.340961 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:11:09.342851 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.322209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:41137)
0619 13:11:09.322918 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.322955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.324672 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.325341 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.325349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.326076 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:09.326732 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.326752 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.328001 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.328006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.329309 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.329323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.330627 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.330636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.330974 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.331823 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:09.331856 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:09.334743 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:09.337979 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:09.337991 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:09.338013 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:09.338674 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:09.339052 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:09.339059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:09.339063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:09.339339 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:09.339847 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:09.339858 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:09.340373 (+ 515us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.340693 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.341277 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260619 13:11:09.344421 16781 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:11:09.344753 16781 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:09.345496 16781 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
I20260619 13:11:09.347154 16464 ts_manager.cc:194] Registered new tserver with Master: f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069)
I20260619 13:11:09.347966 16464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41137
W20260619 13:11:09.671622 16784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:09.671905 16784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:09.672000 16784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:09.679384 16784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:09.679507 16784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:09.679625 16784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:11:09.689833 16784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36461
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:09.691102 16784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:09.692389 16784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:09.705626 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:09.705651 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:09.705715 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:09.706743 16784 server_base.cc:1034] running on GCE node
I20260619 13:11:09.707322 16784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:09.708218 16784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:09.709456 16784 hybrid_clock.cc:648] HybridClock initialized: now 1781874669709411 us; error 55 us; skew 500 ppm
I20260619 13:11:09.709837 16784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:09 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:09.713164 16784 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:11:09.714831 16784 webserver.cc:460] Webserver started at http://127.0.97.3:36461/ using document root <none> and password file <none>
I20260619 13:11:09.715533 16784 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:09.715704 16784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:09.721266 16784 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 13:11:09.724125 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.725507 16784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:11:09.725725 16784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "dcecc7807646a208374cde90c1644566"
server_key_iv: "67328397abd4ec8ac49a15086fc97aa7"
server_key_version: "encryptionkey@0"
I20260619 13:11:09.726753 16784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:09.743773 16784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:09.748395 16784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:09.748703 16784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:09.749763 16784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:09.751096 16784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:09.751227 16784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.751363 16784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:09.751459 16784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:09.785985 16784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37241
I20260619 13:11:09.786074 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37241 every 8 connection(s)
I20260619 13:11:09.787482 16784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:11:09.795207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16784
Jun 19 13:11:09 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:09.809654 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.790229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39731 (local address 127.0.97.62:40341)
0619 13:11:09.790424 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.790428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.791685 (+ 1257us) server_negotiation.cc:407] Connection header received
0619 13:11:09.793549 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.793554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.793692 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.793808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:09.795315 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.796222 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.798584 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.799300 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.799435 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.803609 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:09.803615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:09.803620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:09.803667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:09.806460 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.807250 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.807257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.807261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.807393 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.807848 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.807854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.807858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.808162 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:09.808324 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.809032 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.809256 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":12160}
I20260619 13:11:09.811113 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.790731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:39731)
0619 13:11:09.791495 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.791533 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.793267 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.794054 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.794063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.794571 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:09.795132 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.795150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.796383 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.796392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.798409 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.798427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.799749 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.799759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.800064 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.800898 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:09.800933 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:09.803397 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:09.806632 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:09.806641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:09.806660 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:09.807119 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:09.807511 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:09.807519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:09.807522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:09.807734 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:09.808293 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:09.808302 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:09.808662 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.808937 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.809509 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":337,"threads_started":1}
I20260619 13:11:09.811715 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.796266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.97.62:40341)
0619 13:11:09.797117 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.797121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.797136 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:11:09.797175 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.797179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.797230 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.797316 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:09.799626 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.800790 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.802401 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.802956 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.803152 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.806357 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:09.806362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:09.806365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:09.806413 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:09.809010 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.809738 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.809743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.809747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.809874 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.810291 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.810294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.810296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.810524 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:09.810729 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.811067 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.811252 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":605,"threads_started":1}
I20260619 13:11:09.812974 16912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40341
I20260619 13:11:09.813299 16912 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:09.814087 16912 heartbeater.cc:511] Master 127.0.97.62:40341 requested a full tablet report, sending...
I20260619 13:11:09.815464 16463 ts_manager.cc:194] Registered new tserver with Master: ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
I20260619 13:11:09.816313 16463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39731
I20260619 13:11:09.827219 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:11:09.834426 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.828260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.97.62:40341)
0619 13:11:09.828472 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.828480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.829008 (+ 528us) server_negotiation.cc:407] Connection header received
0619 13:11:09.829267 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.829272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.829445 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.829641 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:11:09.830121 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.830894 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.832317 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.832759 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.832902 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.833449 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:11:09.833725 (+ 276us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 13:11:09.834133 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46388: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 13:11:09.834529 16916 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46388: token expired
W20260619 13:11:09.835067 16387 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:40341)
I20260619 13:11:09.847774 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.835521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.0.97.62:40341)
0619 13:11:09.835709 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.835713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.835729 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:11:09.835842 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.835845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.835900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.835975 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:09.837268 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.838032 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.839443 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.839818 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.839905 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.842762 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:09.842766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:09.842770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:09.842801 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:09.845217 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.845918 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.845922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.845925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.846053 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:09.846375 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:09.846378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:09.846380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:09.846718 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:09.846902 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.847132 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.847293 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:11:09.851161 16463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:11:09.858371 16463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:11:09.893800 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.870939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.97.2:40069)
0619 13:11:09.883550 (+ 12611us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.883564 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.885989 (+ 2425us) server_negotiation.cc:407] Connection header received
0619 13:11:09.886127 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.886139 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.886466 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.886673 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:09.887484 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.889062 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.891325 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.892517 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.892648 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.892851 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.893229 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.893408 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12383,"thread_start_us":12620,"threads_started":1}
I20260619 13:11:09.893893 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.870897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40069 (local address 127.0.0.1:39128)
0619 13:11:09.885366 (+ 14469us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.885410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.885674 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.886695 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.886704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.886763 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:09.887267 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.887286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.889239 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.889244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.891168 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.891180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.892747 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.892753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.892954 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.893005 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.893155 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.893357 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14172,"spinlock_wait_cycles":4736,"thread_start_us":15061,"threads_started":1}
I20260619 13:11:09.894644 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.871084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37241 (local address 127.0.0.1:57018)
0619 13:11:09.886269 (+ 15185us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.886294 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.886449 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.887093 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.887097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.887122 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:09.887553 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.887569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.889581 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.889639 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.891912 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.891927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.893827 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.893836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.894046 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.894077 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.894154 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.894282 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15037,"mutex_wait_us":35,"thread_start_us":15242,"threads_started":1}
I20260619 13:11:09.895687 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.871964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.0.97.3:37241)
0619 13:11:09.884757 (+ 12793us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.884766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.886564 (+ 1798us) server_negotiation.cc:407] Connection header received
0619 13:11:09.886684 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.886691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.886897 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.887099 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:09.887806 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.889434 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.892074 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.893365 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.893486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.895053 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.895129 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.895293 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12607,"thread_start_us":13249,"threads_started":1}
I20260619 13:11:09.895928 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.871435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35138)
0619 13:11:09.886633 (+ 15198us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.886650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.886767 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.887542 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.887547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.887572 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:09.888067 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.888082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.890039 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.890045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.892184 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.892194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.895161 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.895165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.895315 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.895344 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.895442 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.895585 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15033,"spinlock_wait_cycles":2048,"thread_start_us":13501,"threads_started":1}
I20260619 13:11:09.896000 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.872550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.0.97.1:45167)
0619 13:11:09.884278 (+ 11728us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.884287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.886901 (+ 2614us) server_negotiation.cc:407] Connection header received
0619 13:11:09.887010 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.887017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.887236 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.887445 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:09.888244 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.889902 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.892324 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.894683 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.894838 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.895043 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.895515 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.895691 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11564,"thread_start_us":11412,"threads_started":1}
I20260619 13:11:09.896564 16649 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
I20260619 13:11:09.898553 16716 tablet_service.cc:1467] Processing CreateTablet for tablet 36461ee445c44716a9d7782d46181b2c (DEFAULT_TABLE table=test-table [id=848cc8fc2864447193d52b512930e452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:09.899509 16584 tablet_service.cc:1467] Processing CreateTablet for tablet 36461ee445c44716a9d7782d46181b2c (DEFAULT_TABLE table=test-table [id=848cc8fc2864447193d52b512930e452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:09.899526 16847 tablet_service.cc:1467] Processing CreateTablet for tablet 36461ee445c44716a9d7782d46181b2c (DEFAULT_TABLE table=test-table [id=848cc8fc2864447193d52b512930e452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:09.900374 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36461ee445c44716a9d7782d46181b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:09.900802 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36461ee445c44716a9d7782d46181b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:09.900887 16847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36461ee445c44716a9d7782d46181b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:09.913166 16928 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Bootstrap starting.
I20260619 13:11:09.916898 16928 tablet_bootstrap.cc:654] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:09.918035 16928 log.cc:824] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:09.921221 16928 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: No bootstrap required, opened a new log
I20260619 13:11:09.921531 16928 ts_tablet_manager.cc:1397] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 13:11:09.922423 16929 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Bootstrap starting.
I20260619 13:11:09.922820 16930 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Bootstrap starting.
I20260619 13:11:09.926209 16929 tablet_bootstrap.cc:654] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:09.926210 16930 tablet_bootstrap.cc:654] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:09.927573 16930 log.cc:824] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:09.927572 16929 log.cc:824] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:09.930347 16928 raft_consensus.cc:348] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.930874 16928 raft_consensus.cc:374] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:09.931038 16930 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: No bootstrap required, opened a new log
I20260619 13:11:09.931061 16928 raft_consensus.cc:729] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceaa825ea4f3431e96b0d1a52db549f0, State: Initialized, Role: FOLLOWER
I20260619 13:11:09.931309 16929 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: No bootstrap required, opened a new log
I20260619 13:11:09.931377 16930 ts_tablet_manager.cc:1397] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 13:11:09.931573 16929 ts_tablet_manager.cc:1397] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:11:09.931865 16928 consensus_queue.cc:260] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.933696 16928 ts_tablet_manager.cc:1428] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 13:11:09.940037 16930 raft_consensus.cc:348] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.940521 16930 raft_consensus.cc:374] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:09.940629 16930 raft_consensus.cc:729] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f695a0054ed44edb8e7dfcb6aac63713, State: Initialized, Role: FOLLOWER
I20260619 13:11:09.941241 16930 consensus_queue.cc:260] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.941334 16929 raft_consensus.cc:348] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.941794 16929 raft_consensus.cc:374] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:09.941908 16929 raft_consensus.cc:729] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebeab24bbcf24e7cb6e40d7a071f8e84, State: Initialized, Role: FOLLOWER
I20260619 13:11:09.942469 16781 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
I20260619 13:11:09.942510 16929 consensus_queue.cc:260] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.943181 16930 ts_tablet_manager.cc:1428] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 13:11:09.943869 16912 heartbeater.cc:503] Master 127.0.97.62:40341 was elected leader, sending a full tablet report...
I20260619 13:11:09.944211 16929 ts_tablet_manager.cc:1428] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 13:11:09.976133 16934 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:09.976473 16934 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.978374 16934 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
I20260619 13:11:09.985446 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.978831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37201 (local address 127.0.97.3:37241)
0619 13:11:09.979050 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.979059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.979800 (+ 741us) server_negotiation.cc:407] Connection header received
0619 13:11:09.980077 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.980081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.980161 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.980251 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:09.980975 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.981723 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.983616 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.984435 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.984573 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.984703 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.985047 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.985179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:11:09.985477 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.978874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47011 (local address 127.0.97.2:40069)
0619 13:11:09.979136 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:09.979145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:09.979739 (+ 594us) server_negotiation.cc:407] Connection header received
0619 13:11:09.979993 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:09.980002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:09.980090 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:09.980165 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:09.980868 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.981693 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.983673 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.984439 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.984600 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.984709 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:09.985083 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 13:11:09.985215 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:11:09.985659 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.978523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40069 (local address 127.0.97.1:47011)
0619 13:11:09.979619 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.979635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.979812 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.980215 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.980221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.980248 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:09.980717 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.980730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.981819 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.981828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.983483 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.983495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.984700 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.984706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.984902 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.984925 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.985035 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.985171 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":141,"spinlock_wait_cycles":28032,"thread_start_us":872,"threads_started":1}
I20260619 13:11:09.985998 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:09.978840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37241 (local address 127.0.97.1:37201)
0619 13:11:09.979679 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:09.979693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:09.979936 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:09.980286 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:09.980294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:09.980324 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:09.980823 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.980837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.981866 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.981870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.983470 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:09.983482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:09.984684 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:09.984691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:09.984865 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:09.984913 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 13:11:09.984988 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 13:11:09.985142 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":29,"thread_start_us":430,"threads_started":1}
I20260619 13:11:09.986794 16736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36461ee445c44716a9d7782d46181b2c" candidate_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f695a0054ed44edb8e7dfcb6aac63713" is_pre_election: true
I20260619 13:11:09.986927 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36461ee445c44716a9d7782d46181b2c" candidate_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" is_pre_election: true
I20260619 13:11:09.987349 16736 raft_consensus.cc:2454] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceaa825ea4f3431e96b0d1a52db549f0 in term 0.
I20260619 13:11:09.987422 16867 raft_consensus.cc:2454] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceaa825ea4f3431e96b0d1a52db549f0 in term 0.
I20260619 13:11:09.988162 16540 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0, ebeab24bbcf24e7cb6e40d7a071f8e84; no voters:
I20260619 13:11:09.988583 16934 raft_consensus.cc:2781] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:11:09.988736 16934 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:11:09.988865 16934 raft_consensus.cc:3037] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:09.991281 16934 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:09.991981 16934 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 1 election: Requested vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
I20260619 13:11:09.992542 16736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36461ee445c44716a9d7782d46181b2c" candidate_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f695a0054ed44edb8e7dfcb6aac63713"
I20260619 13:11:09.992543 16867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36461ee445c44716a9d7782d46181b2c" candidate_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
I20260619 13:11:09.992812 16736 raft_consensus.cc:3037] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:09.992825 16867 raft_consensus.cc:3037] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:09.995249 16867 raft_consensus.cc:2454] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceaa825ea4f3431e96b0d1a52db549f0 in term 1.
I20260619 13:11:09.995410 16736 raft_consensus.cc:2454] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceaa825ea4f3431e96b0d1a52db549f0 in term 1.
I20260619 13:11:09.995821 16540 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0, ebeab24bbcf24e7cb6e40d7a071f8e84; no voters:
I20260619 13:11:09.996240 16934 raft_consensus.cc:2781] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:09.996728 16934 raft_consensus.cc:686] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 LEADER]: Becoming Leader. State: Replica: ceaa825ea4f3431e96b0d1a52db549f0, State: Running, Role: LEADER
I20260619 13:11:09.997313 16934 consensus_queue.cc:237] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:10.002333 16464 catalog_manager.cc:5495] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } health_report { overall_health: HEALTHY } } }
W20260619 13:11:10.039265 16913 tablet.cc:2369] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:10.043984 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.038268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.97.1:45167)
0619 13:11:10.038533 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:10.038539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:10.038646 (+ 107us) server_negotiation.cc:407] Connection header received
0619 13:11:10.038939 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:10.038944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:10.039023 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:10.039190 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:11:10.039592 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.040352 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.041781 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.042185 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.042323 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:10.042869 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:11:10.043359 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:11:10.043524 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:10.043613 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 13:11:10.043698 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 13:11:10.053104 16867 raft_consensus.cc:1261] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Refusing update from remote peer ceaa825ea4f3431e96b0d1a52db549f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:10.053115 16736 raft_consensus.cc:1261] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Refusing update from remote peer ceaa825ea4f3431e96b0d1a52db549f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:10.054399 16939 consensus_queue.cc:1035] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:10.054809 16934 consensus_queue.cc:1035] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:10.067806 16942 mvcc.cc:205] Tried to move back new op lower bound from 7298558648525307904 to 7298558648313180160. Current Snapshot: MvccSnapshot[applied={T|T < 7298558648525307904}]
I20260619 13:11:10.085740 16464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:10.086062 16464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:10.090299 16464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Sending DeleteTablet for 3 replicas of tablet 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:10.091712 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 36461ee445c44716a9d7782d46181b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260619 13:11:10.091681 16716 tablet_service.cc:1514] Processing DeleteTablet for tablet 36461ee445c44716a9d7782d46181b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39128
I20260619 13:11:10.091729 16847 tablet_service.cc:1514] Processing DeleteTablet for tablet 36461ee445c44716a9d7782d46181b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57018
I20260619 13:11:10.093393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16501
I20260619 13:11:10.121440 16954 tablet_replica.cc:330] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: stopping tablet replica
I20260619 13:11:10.122293 16954 raft_consensus.cc:2229] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:10.123154 16954 raft_consensus.cc:2258] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:10.124500 16952 tablet_replica.cc:330] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: stopping tablet replica
I20260619 13:11:10.124712 16953 tablet_replica.cc:330] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: stopping tablet replica
I20260619 13:11:10.125183 16953 raft_consensus.cc:2229] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:11:10.126065 16953 raft_consensus.cc:2258] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:10.126392 16954 ts_tablet_manager.cc:1905] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:10.126432 16952 raft_consensus.cc:2229] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:10.127019 16952 raft_consensus.cc:2258] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:10.129673 16952 ts_tablet_manager.cc:1905] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:10.130895 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.094446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.97.1:45167)
0619 13:11:10.094699 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:10.094709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:10.094890 (+ 181us) server_negotiation.cc:407] Connection header received
0619 13:11:10.095138 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:10.095146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:10.095253 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:10.095376 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:10.098003 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.099917 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.104166 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.105199 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.105382 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:10.115560 (+ 10178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:10.115584 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:10.115611 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:10.115691 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:10.126283 (+ 10592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.127452 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.127463 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.127468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.127692 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.128226 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.128231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.128234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.128721 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:10.129025 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:10.130181 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0619 13:11:10.130459 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:11:10.131748 16953 ts_tablet_manager.cc:1905] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:10.132135 16644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:10.134869 16954 ts_tablet_manager.cc:1918] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:10.135113 16954 log.cc:1196] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/36461ee445c44716a9d7782d46181b2c
I20260619 13:11:10.135915 16954 ts_tablet_manager.cc:1939] T 36461ee445c44716a9d7782d46181b2c P ebeab24bbcf24e7cb6e40d7a071f8e84: Deleting consensus metadata
I20260619 13:11:10.138255 16451 catalog_manager.cc:4841] TS ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): tablet 36461ee445c44716a9d7782d46181b2c (table test-table [id=848cc8fc2864447193d52b512930e452]) successfully deleted
I20260619 13:11:10.138849 16952 ts_tablet_manager.cc:1918] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:10.139020 16952 log.cc:1196] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/36461ee445c44716a9d7782d46181b2c
I20260619 13:11:10.139497 16952 ts_tablet_manager.cc:1939] T 36461ee445c44716a9d7782d46181b2c P f695a0054ed44edb8e7dfcb6aac63713: Deleting consensus metadata
I20260619 13:11:10.141054 16449 catalog_manager.cc:4841] TS f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): tablet 36461ee445c44716a9d7782d46181b2c (table test-table [id=848cc8fc2864447193d52b512930e452]) successfully deleted
I20260619 13:11:10.233742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16501
I20260619 13:11:10.252334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16653
W20260619 13:11:10.252492 16450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45167 (error 108)
I20260619 13:11:10.254408 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.253994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35168)
0619 13:11:10.254161 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.254271 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 13:11:10.254657 16450 catalog_manager.cc:4584] TS ceaa825ea4f3431e96b0d1a52db549f0 (127.0.97.1:45167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36461ee445c44716a9d7782d46181b2c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
I20260619 13:11:10.270800 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.253159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.97.2:40069)
0619 13:11:10.253345 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:10.253351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:10.253440 (+ 89us) server_negotiation.cc:407] Connection header received
0619 13:11:10.253736 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:10.253748 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:10.253816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:10.253932 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:10.255462 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.256188 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.257443 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.257918 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.258058 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:10.264955 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:10.264972 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:10.264988 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:10.265040 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:10.267872 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.268621 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.268636 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.268639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.268774 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.269165 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.269177 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.269180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.269552 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:10.269929 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:10.270233 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 13:11:10.270453 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:11:10.271456 16776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:10.308478 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.307967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35184)
0619 13:11:10.308198 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.308313 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 13:11:10.309746 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.309314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35188)
0619 13:11:10.309472 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.309548 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:11:10.362962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16653
I20260619 13:11:10.370296 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.369695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35194)
0619 13:11:10.369946 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.370076 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 13:11:10.371464 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.370984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35196)
0619 13:11:10.371211 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.371311 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 13:11:10.380959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16784
I20260619 13:11:10.394223 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.381791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.97.3:37241)
0619 13:11:10.381972 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:10.381976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:10.381995 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:11:10.382100 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:10.382104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:10.382166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:10.382249 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:10.383542 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.384268 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.385578 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:10.386017 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:10.386113 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:10.388552 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:10.388567 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:10.388583 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:10.388631 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:10.391386 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.392026 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.392033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.392035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.392171 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:10.392606 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:10.392611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:10.392613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:10.393020 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:10.393278 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:10.393676 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 13:11:10.393871 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:11:10.394963 16907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:10.452029 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.451512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35206)
0619 13:11:10.451773 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.451865 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 13:11:10.453401 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:10.452906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45167 (local address 127.0.0.1:35210)
0619 13:11:10.453165 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:10.453268 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 13:11:10.484680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16784
I20260619 13:11:10.501236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 16431
I20260619 13:11:10.502516 16494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:10.620151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 16431
I20260619 13:11:30.642338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--webserver_interface=127.0.97.62
--webserver_port=42995
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:30.973975 16960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:30.974277 16960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:30.974380 16960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:30.981500 16960 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:11:30.981663 16960 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:11:30.981729 16960 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:11:30.981786 16960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:11:30.981839 16960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:30.981907 16960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:30.981953 16960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:11:30.982007 16960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:11:30.992077 16960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:30.993321 16960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:30.994709 16960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:31.007076 16965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:31.007100 16968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:31.007074 16966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:31.007601 16960 server_base.cc:1034] running on GCE node
I20260619 13:11:31.008723 16960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:31.010305 16960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:31.011567 16960 hybrid_clock.cc:648] HybridClock initialized: now 1781874691011487 us; error 57 us; skew 500 ppm
I20260619 13:11:31.011922 16960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:31 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:31.015357 16960 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:11:31.016849 16960 webserver.cc:460] Webserver started at http://127.0.97.62:42995/ using document root <none> and password file <none>
I20260619 13:11:31.017553 16960 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:31.017724 16960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:31.023240 16960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:11:31.026278 16975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:31.027609 16960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:11:31.027778 16960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "965325d4ae084620994beb277dbae21d"
format_stamp: "Formatted at 2026-06-19 13:10:44 on dist-test-slave-q11v"
server_key: "4778f4613ba62a3daae72b07ca8a3922"
server_key_iv: "1b5d0c400da564a7f8bd0fecf7222556"
server_key_version: "encryptionkey@0"
I20260619 13:11:31.028765 16960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:31.043601 16960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:31.048108 16960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:31.048453 16960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:31.069577 16960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40341
I20260619 13:11:31.069648 17026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40341 every 8 connection(s)
I20260619 13:11:31.071450 16960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:11:31.076333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 16960
I20260619 13:11:31.077270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45167
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=38117
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:11:31.078909 17027 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:11:31.082454 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap starting.
I20260619 13:11:31.103266 17027 log.cc:824] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:31.119490 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:11:31.120077 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Bootstrap complete.
I20260619 13:11:31.130213 17027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:31.131292 17027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965325d4ae084620994beb277dbae21d, State: Initialized, Role: FOLLOWER
I20260619 13:11:31.132017 17027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:31.132325 17027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:11:31.132490 17027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:11:31.132668 17027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:11:31.136811 17027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:31.137380 17027 leader_election.cc:302] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965325d4ae084620994beb277dbae21d; no voters:
I20260619 13:11:31.138115 17027 leader_election.cc:288] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:11:31.138259 17031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:11:31.139576 17031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [term 3 LEADER]: Becoming Leader. State: Replica: 965325d4ae084620994beb277dbae21d, State: Running, Role: LEADER
I20260619 13:11:31.140199 17031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } }
I20260619 13:11:31.140493 17027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:11:31.143085 17032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:11:31.143527 17032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:31.144302 17033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965325d4ae084620994beb277dbae21d. Latest consensus state: current_term: 3 leader_uuid: "965325d4ae084620994beb277dbae21d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965325d4ae084620994beb277dbae21d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40341 } } }
I20260619 13:11:31.144570 17037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:11:31.144618 17033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:31.155843 17037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:11:31.157905 17037 catalog_manager.cc:1259] Loaded cluster ID: 3e57a68d67a64efa94f4b2a82c8b5efb
I20260619 13:11:31.158072 17037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:11:31.163658 17037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:11:31.169917 17037 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Loaded TSK: 0
I20260619 13:11:31.170344 17037 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 965325d4ae084620994beb277dbae21d: Loaded TSK: 1
F20260619 13:11:31.171923 17037 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd4320159ad google::LogMessage::Fail() at ??:0
@ 0x7fd43201888a google::LogMessage::SendToLog() at ??:0
@ 0x7fd4320154de google::LogMessage::Flush() at ??:0
@ 0x7fd432016a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd435d797d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd435d6873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd447a441c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd447a43783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd447acf2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd447acf0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd447b195e9 std::function<>::operator()() at ??:0
@ 0x7fd447a47cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd447a45c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd447acd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd447acd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd434c61345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd434c6491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd434c64799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd434c3feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd437f196db start_thread at ??:0
@ 0x7fd430eaf61f clone at ??:0
W20260619 13:11:31.480099 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:31.480464 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:31.480635 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:31.488098 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:31.488276 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:31.488456 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:11:31.499368 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38117
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:31.500857 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:31.502319 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:31.516266 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:31.516412 17076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:31.516248 17078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:31.517189 17029 server_base.cc:1034] running on GCE node
I20260619 13:11:31.517951 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:31.518913 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:31.520125 17029 hybrid_clock.cc:648] HybridClock initialized: now 1781874691520075 us; error 36 us; skew 500 ppm
I20260619 13:11:31.520465 17029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:31 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:31.523814 17029 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:11:31.525334 17029 webserver.cc:460] Webserver started at http://127.0.97.1:38117/ using document root <none> and password file <none>
I20260619 13:11:31.526202 17029 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:31.526343 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:31.532265 17029 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 13:11:31.535533 17085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:31.537007 17029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:11:31.537182 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "ceaa825ea4f3431e96b0d1a52db549f0"
format_stamp: "Formatted at 2026-06-19 13:10:45 on dist-test-slave-q11v"
server_key: "aef74954e13da16eec6bc89fc1de1625"
server_key_iv: "99901fcf7638aaf86a01a5c3b404dce1"
server_key_version: "encryptionkey@0"
I20260619 13:11:31.538283 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:31.562143 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:31.566744 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:31.567046 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:31.568117 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:31.570139 17092 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 13:11:31.576090 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 13:11:31.576241 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 13:11:31.576393 17029 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 13:11:31.579795 17029 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 13:11:31.579892 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260619 13:11:31.580314 17092 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Bootstrap starting.
I20260619 13:11:31.589394 17092 log.cc:824] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:31.606295 17092 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:11:31.607230 17092 tablet_bootstrap.cc:492] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Bootstrap complete.
I20260619 13:11:31.607875 17092 ts_tablet_manager.cc:1397] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20260619 13:11:31.618021 17092 raft_consensus.cc:348] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:31.619591 17092 raft_consensus.cc:729] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceaa825ea4f3431e96b0d1a52db549f0, State: Initialized, Role: FOLLOWER
I20260619 13:11:31.620393 17092 consensus_queue.cc:260] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:31.622035 17092 ts_tablet_manager.cc:1428] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260619 13:11:31.623862 17029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45167
I20260619 13:11:31.623899 17200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45167 every 8 connection(s)
I20260619 13:11:31.625116 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:11:31.629878 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:31.627916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:56889)
0619 13:11:31.628699 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:31.628888 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":326,"threads_started":1}
W20260619 13:11:31.630522 17201 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40341 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40341: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
I20260619 13:11:31.631574 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:31.631206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:52937)
0619 13:11:31.631377 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:31.631466 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 13:11:31.632516 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:31.632179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:35351)
0619 13:11:31.632316 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:31.632385 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:11:31.632758 17201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:11:31.633308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17029
I20260619 13:11:31.633926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40069
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=33329
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:31.969862 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:31.970201 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:31.970372 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:31.977842 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:31.978003 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:31.978180 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:11:31.988286 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33329
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:31.989667 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:31.991055 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:32.005120 17209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:32.005125 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:32.005116 17212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:32.006651 17204 server_base.cc:1034] running on GCE node
I20260619 13:11:32.007262 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:32.008235 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:32.009443 17204 hybrid_clock.cc:648] HybridClock initialized: now 1781874692009397 us; error 34 us; skew 500 ppm
I20260619 13:11:32.009824 17204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:32 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:32.013008 17204 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:11:32.014570 17204 webserver.cc:460] Webserver started at http://127.0.97.2:33329/ using document root <none> and password file <none>
I20260619 13:11:32.015286 17204 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:32.015429 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:32.020922 17204 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 13:11:32.023816 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.025228 17204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:11:32.025408 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "f695a0054ed44edb8e7dfcb6aac63713"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "ee20942e41dd2abeb1c2693b13abd8dc"
server_key_iv: "3110c2e25ff1d962ed4fcdadbe3fbc3e"
server_key_version: "encryptionkey@0"
I20260619 13:11:32.026497 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:32.044082 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:32.048360 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:32.048642 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:32.049681 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:32.051010 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:32.051137 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.051265 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:32.051353 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.084684 17204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40069
I20260619 13:11:32.084734 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40069 every 8 connection(s)
I20260619 13:11:32.086298 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:11:32.087421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17204
I20260619 13:11:32.088145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37241
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40341
--webserver_port=36461
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:11:32.091722 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.089528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:57437)
0619 13:11:32.090467 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.090728 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":362,"threads_started":1}
W20260619 13:11:32.092567 17332 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40341 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40341: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
I20260619 13:11:32.093454 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.093116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:35633)
0619 13:11:32.093271 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.093350 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:11:32.094317 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.093987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:42827)
0619 13:11:32.094124 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.094209 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:11:32.094647 17332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:11:32.424708 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:32.425004 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:32.425186 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:32.432682 17335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:32.432814 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:32.432997 17335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:11:32.443506 17335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36461
--tserver_master_addrs=127.0.97.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:32.444976 17335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:32.446393 17335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:32.459357 17343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:32.459513 17340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:32.459460 17341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:32.460494 17335 server_base.cc:1034] running on GCE node
I20260619 13:11:32.461123 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:32.462220 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:32.463428 17335 hybrid_clock.cc:648] HybridClock initialized: now 1781874692463361 us; error 57 us; skew 500 ppm
I20260619 13:11:32.463766 17335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:32 dist-test-slave-q11v krb5kdc[15882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:32.466982 17335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:11:32.468539 17335 webserver.cc:460] Webserver started at http://127.0.97.3:36461/ using document root <none> and password file <none>
I20260619 13:11:32.469252 17335 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:32.469396 17335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:32.474936 17335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:11:32.477885 17350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.479281 17335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:11:32.479470 17335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84"
format_stamp: "Formatted at 2026-06-19 13:10:46 on dist-test-slave-q11v"
server_key: "dcecc7807646a208374cde90c1644566"
server_key_iv: "67328397abd4ec8ac49a15086fc97aa7"
server_key_version: "encryptionkey@0"
I20260619 13:11:32.480518 17335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:32.514628 17335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:32.518778 17335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:32.519079 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:32.520098 17335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:32.521543 17335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:32.521682 17335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.521811 17335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:32.521904 17335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:32.553046 17335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37241
I20260619 13:11:32.553157 17462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37241 every 8 connection(s)
I20260619 13:11:32.554581 17335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:11:32.559474 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.557481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:60835)
0619 13:11:32.558375 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.558581 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":368,"threads_started":1}
W20260619 13:11:32.560149 17463 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40341 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40341: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
I20260619 13:11:32.560935 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.560640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:53633)
0619 13:11:32.560769 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.560832 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 13:11:32.561664 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.561375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:48763)
0619 13:11:32.561481 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.561547 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 13:11:32.561903 17463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:11:32.562193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17335
I20260619 13:11:32.634073 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:32.633266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:52829)
0619 13:11:32.633770 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:32.633863 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":211,"threads_started":1}
I20260619 13:11:33.095960 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.095182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:46559)
0619 13:11:33.095638 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:33.095727 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":216,"threads_started":1}
I20260619 13:11:33.326324 17469 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:33.326614 17469 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:33.328260 17469 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
Jun 19 13:11:33 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 19 13:11:33 dist-test-slave-q11v krb5kdc[15882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:11:33.352913 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.328629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37241 (local address 127.0.97.1:58349)
0619 13:11:33.330877 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:33.330900 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:33.331060 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:33.334414 (+ 3354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:33.334427 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:33.335263 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:33.335945 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:33.335966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.337575 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.337579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:33.339162 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:33.339175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.340196 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.340202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:33.340526 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:33.341499 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:33.341526 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:33.344712 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:33.348481 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:33.348492 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:33.348510 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:33.349213 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:33.349814 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:33.349817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:33.349819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:33.350036 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:33.351076 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:33.351085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:33.351471 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 13:11:33.351812 (+ 341us) client_negotiation.cc:240] Negotiation successful
0619 13:11:33.352379 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2084,"mutex_wait_us":172,"spinlock_wait_cycles":28800,"thread_start_us":1913,"threads_started":1}
I20260619 13:11:33.353042 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.328934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58349 (local address 127.0.97.3:37241)
0619 13:11:33.331818 (+ 2884us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:33.331830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:33.331861 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:11:33.333330 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:33.333365 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:33.333886 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:33.334259 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:33.336170 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.337372 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.339310 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.339990 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.340198 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:33.345079 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:33.345091 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:33.345110 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:33.345171 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:33.348319 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:33.349400 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:33.349408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:33.349416 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:33.349693 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:33.350159 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:33.350162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:33.350164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:33.350878 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:33.351018 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:33.352169 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0619 13:11:33.352629 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2726,"thread_start_us":2604,"threads_started":1}
I20260619 13:11:33.353449 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.328929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57347 (local address 127.0.97.2:40069)
0619 13:11:33.331515 (+ 2586us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:33.331527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:33.331568 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:11:33.333358 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:33.333367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:33.333905 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:33.334759 (+ 854us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:33.336162 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.337438 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.339332 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.340029 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.340250 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:33.345108 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:33.345124 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:33.345138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:33.345183 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:33.349286 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:33.350117 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:33.350127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:33.350134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:33.350324 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:33.350773 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:33.350777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:33.350779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:33.351483 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:33.351724 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:33.352250 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 13:11:33.352688 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2404,"thread_start_us":2160,"threads_started":1}
I20260619 13:11:33.353641 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.328485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40069 (local address 127.0.97.1:57347)
0619 13:11:33.329134 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:33.329175 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:33.329888 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:33.334704 (+ 4816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:33.334709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:33.335439 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:33.335945 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:33.335966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.337644 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.337650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:33.339172 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:33.339186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:33.340244 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:33.340252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:33.340529 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:33.341560 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:33.341578 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:33.344803 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:33.349420 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:33.349428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:33.349431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:33.349966 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:33.350440 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:33.350445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:33.350448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:33.350616 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:33.351657 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:33.351661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:33.351822 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 13:11:33.351977 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:11:33.352576 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":290,"thread_start_us":642,"threads_started":1}
W20260619 13:11:33.356303 17087 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
W20260619 13:11:33.356576 17089 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:33.356770 17089 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0; no voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713
I20260619 13:11:33.357452 17469 raft_consensus.cc:2726] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:11:33.563328 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.562418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:54343)
0619 13:11:33.562907 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:33.563035 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260619 13:11:33.635484 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:33.635064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:40117)
0619 13:11:33.635243 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:33.635341 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 13:11:34.097733 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:34.096911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:54079)
0619 13:11:34.097425 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:34.097503 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":300,"threads_started":1}
I20260619 13:11:34.564916 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:34.564260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:60455)
0619 13:11:34.564681 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:34.564761 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260619 13:11:34.637228 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:34.636427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:59497)
0619 13:11:34.636932 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:34.637027 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260619 13:11:35.099350 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:35.098608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:56367)
0619 13:11:35.099074 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:35.099180 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":276,"threads_started":1}
I20260619 13:11:35.566846 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:35.566049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:51891)
0619 13:11:35.566549 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:35.566643 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260619 13:11:35.580880 17480 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:35.581058 17480 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:35.581722 17480 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
W20260619 13:11:35.582737 17087 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
W20260619 13:11:35.582929 17089 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:35.583065 17089 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0; no voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713
I20260619 13:11:35.583312 17480 raft_consensus.cc:2726] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:11:35.639004 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:35.638223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:57287)
0619 13:11:35.638716 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:35.638814 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":229,"threads_started":1}
I20260619 13:11:36.101127 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:36.100305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:47473)
0619 13:11:36.100771 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:36.100897 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260619 13:11:36.568553 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:36.567828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:34033)
0619 13:11:36.568302 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:36.568381 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":231,"threads_started":1}
I20260619 13:11:36.640833 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:36.640074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:60249)
0619 13:11:36.640551 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:36.640653 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":230,"threads_started":1}
I20260619 13:11:37.102860 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:37.102132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:56109)
0619 13:11:37.102591 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:37.102678 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260619 13:11:37.570320 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:37.569524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:51171)
0619 13:11:37.570058 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:37.570164 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":291,"threads_started":1}
I20260619 13:11:37.642552 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:37.641835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:35421)
0619 13:11:37.642305 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:37.642387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
I20260619 13:11:38.020634 17488 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:38.020844 17488 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:38.021628 17488 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
W20260619 13:11:38.022575 17087 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
W20260619 13:11:38.022827 17089 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:38.022959 17089 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0; no voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713
I20260619 13:11:38.023206 17488 raft_consensus.cc:2726] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:11:38.104628 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:38.103867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:42309)
0619 13:11:38.104346 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:38.104433 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260619 13:11:38.572119 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:38.571308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:55815)
0619 13:11:38.571803 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:38.571888 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260619 13:11:38.644253 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:38.643488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:46801)
0619 13:11:38.643931 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:38.644030 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260619 13:11:39.106693 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:39.105781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:45007)
0619 13:11:39.106369 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:39.106487 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":274,"threads_started":1}
I20260619 13:11:39.574105 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:39.573164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:60701)
0619 13:11:39.573778 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:39.573878 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":265,"threads_started":1}
I20260619 13:11:39.645913 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:39.645205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:38667)
0619 13:11:39.645667 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:39.645752 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
I20260619 13:11:40.108870 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:40.107992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:41127)
0619 13:11:40.108556 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:40.108681 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":244,"threads_started":1}
I20260619 13:11:40.575834 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:40.575115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:46239)
0619 13:11:40.575556 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:40.575643 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260619 13:11:40.647583 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:40.646810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:57257)
0619 13:11:40.647285 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:40.647376 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260619 13:11:41.111053 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:41.109883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:54351)
0619 13:11:41.110625 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:41.110779 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":318,"threads_started":1}
I20260619 13:11:41.364290 17499 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:41.364473 17499 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:41.365335 17499 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
W20260619 13:11:41.366413 17087 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
W20260619 13:11:41.366621 17089 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:41.366755 17089 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0; no voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713
I20260619 13:11:41.367055 17499 raft_consensus.cc:2726] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:11:41.577732 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:41.576878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:34089)
0619 13:11:41.577421 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:41.577509 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260619 13:11:41.649421 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:41.648656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:33441)
0619 13:11:41.649143 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:41.649221 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260619 13:11:42.112950 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:42.112211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:55581)
0619 13:11:42.112688 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:42.112770 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
I20260619 13:11:42.579679 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:42.578789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:45561)
0619 13:11:42.579378 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:42.579482 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":250,"threads_started":1}
I20260619 13:11:42.651237 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:42.650379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:32791)
0619 13:11:42.650923 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:42.651027 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":294,"threads_started":1}
I20260619 13:11:43.114786 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:43.113982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:49295)
0619 13:11:43.114518 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:43.114610 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260619 13:11:43.581406 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:43.580650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:36589)
0619 13:11:43.581123 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:43.581221 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260619 13:11:43.652935 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:43.652191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:56185)
0619 13:11:43.652664 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:43.652755 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260619 13:11:44.116497 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:44.115755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:50395)
0619 13:11:44.116223 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:44.116302 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":242,"threads_started":1}
I20260619 13:11:44.583114 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:44.582389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:33461)
0619 13:11:44.582857 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:44.582938 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260619 13:11:44.654733 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:44.653974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:52749)
0619 13:11:44.654440 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:44.654528 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260619 13:11:45.118156 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:45.117424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:58077)
0619 13:11:45.117914 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:45.117997 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260619 13:11:45.585147 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:45.584237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:59117)
0619 13:11:45.584837 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:45.584949 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":362,"threads_started":1}
I20260619 13:11:45.644057 17513 raft_consensus.cc:482] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:45.644274 17513 raft_consensus.cc:504] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f695a0054ed44edb8e7dfcb6aac63713" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40069 } } peers { permanent_uuid: "ebeab24bbcf24e7cb6e40d7a071f8e84" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37241 } } peers { permanent_uuid: "ceaa825ea4f3431e96b0d1a52db549f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45167 } }
I20260619 13:11:45.644901 17513 leader_election.cc:288] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069), ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241)
W20260619 13:11:45.645872 17087 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f695a0054ed44edb8e7dfcb6aac63713 (127.0.97.2:40069): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
W20260619 13:11:45.646036 17089 leader_election.cc:341] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebeab24bbcf24e7cb6e40d7a071f8e84 (127.0.97.3:37241): Not found: Tablet not found: 36461ee445c44716a9d7782d46181b2c
I20260619 13:11:45.646191 17089 leader_election.cc:302] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceaa825ea4f3431e96b0d1a52db549f0; no voters: ebeab24bbcf24e7cb6e40d7a071f8e84, f695a0054ed44edb8e7dfcb6aac63713
I20260619 13:11:45.646507 17513 raft_consensus.cc:2726] T 36461ee445c44716a9d7782d46181b2c P ceaa825ea4f3431e96b0d1a52db549f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:11:45.656639 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:45.655707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:41197)
0619 13:11:45.656235 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:45.656348 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
I20260619 13:11:46.119930 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:46.119151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:43199)
0619 13:11:46.119642 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:46.119728 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260619 13:11:46.586854 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:46.586116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:51847)
0619 13:11:46.586572 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:46.586654 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260619 13:11:46.658710 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:46.657988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:43003)
0619 13:11:46.658455 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:46.658536 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":232,"threads_started":1}
I20260619 13:11:47.122007 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.121185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.2:50537)
0619 13:11:47.121749 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:47.121847 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":248,"threads_started":1}
I20260619 13:11:47.573269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17029
I20260619 13:11:47.588896 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.588114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.3:43751)
0619 13:11:47.588608 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:47.588726 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260619 13:11:47.594267 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.574407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.0.97.1:45167)
0619 13:11:47.575129 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:47.575136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:47.575163 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:11:47.575254 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:47.575265 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:47.575611 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:47.575791 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:47.577301 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.578389 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.579774 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.580445 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.580573 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:47.588210 (+ 7637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:47.588234 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:47.588258 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:47.588320 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:47.591592 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.592323 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.592331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.592334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.592491 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.592834 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.592837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.592839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.593214 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:47.593437 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:47.593783 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 13:11:47.593983 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":364,"threads_started":1}
I20260619 13:11:47.595582 17195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:47.686228 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.685103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40341 (local address 127.0.97.1:34939)
0619 13:11:47.685768 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:47.685899 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":304,"threads_started":1}
I20260619 13:11:47.686376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17029
I20260619 13:11:47.704905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17204
I20260619 13:11:47.718712 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.705846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.97.2:40069)
0619 13:11:47.706362 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:47.706367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:47.706383 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:11:47.706440 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:47.706443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:47.706511 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:47.706608 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:47.707906 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.708735 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.710196 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.710627 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.710725 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:47.713333 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:47.713338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:47.713341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:47.713376 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:47.716016 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.716649 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.716653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.716655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.716772 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.717194 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.717197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.717199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.717461 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:47.717732 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:47.718139 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 13:11:47.718329 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260619 13:11:47.719547 17327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:47.804018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17204
I20260619 13:11:47.822367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17335
I20260619 13:11:47.836107 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:47.823302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.0.97.3:37241)
0619 13:11:47.823901 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:47.823905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:47.823921 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:11:47.823974 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:47.823978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:47.824035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:47.824163 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:47.825532 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.826496 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.827726 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:47.828131 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:47.828265 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:47.830869 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:47.830874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:47.830877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:47.830909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:47.833401 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.834154 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.834159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.834163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.834318 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:47.834687 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:47.834694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:47.834697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:47.834961 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:47.835230 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:47.835504 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 13:11:47.835724 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":307,"threads_started":1}
I20260619 13:11:47.836886 17458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:47.921244 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17335
2026-06-19T13:11:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64659 ms)
[----------] 30 tests from SecurityITest (302541 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:11:48 dist-test-slave-q11v krb5kdc[17529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:11:48 dist-test-slave-q11v krb5kdc[17529](info): set up 2 sockets
Jun 19 13:11:48 dist-test-slave-q11v krb5kdc[17529](info): commencing operation
krb5kdc: starting...
W20260619 13:11:49.049887 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:11:49 dist-test-slave-q11v krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:11:49.305290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37453
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:49.641429 17542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:49.641803 17542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:49.641974 17542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:49.649161 17542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:11:49.649326 17542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:49.649446 17542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:49.649557 17542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:11:49.649655 17542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:11:49.659548 17542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:49.660962 17542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:49.662390 17542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:49.674933 17550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:49.674902 17547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:49.675000 17548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:49.676365 17542 server_base.cc:1034] running on GCE node
I20260619 13:11:49.677114 17542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:49.678602 17542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:49.679808 17542 hybrid_clock.cc:648] HybridClock initialized: now 1781874709679767 us; error 31 us; skew 500 ppm
I20260619 13:11:49.680173 17542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:49 dist-test-slave-q11v krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:49.683298 17542 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:11:49.684870 17542 webserver.cc:460] Webserver started at http://127.0.97.62:37365/ using document root <none> and password file <none>
I20260619 13:11:49.685578 17542 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:49.685794 17542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:49.686115 17542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:49.689158 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "f79ba94694cb4b82ad855dcc057c8c05"
format_stamp: "Formatted at 2026-06-19 13:11:49 on dist-test-slave-q11v"
server_key: "11cddab673ae70344552bd2e0ba7c81d"
server_key_iv: "7ae60f4ed3eb41c91e7bd739999fd1de"
server_key_version: "encryptionkey@0"
I20260619 13:11:49.690017 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "f79ba94694cb4b82ad855dcc057c8c05"
format_stamp: "Formatted at 2026-06-19 13:11:49 on dist-test-slave-q11v"
server_key: "11cddab673ae70344552bd2e0ba7c81d"
server_key_iv: "7ae60f4ed3eb41c91e7bd739999fd1de"
server_key_version: "encryptionkey@0"
I20260619 13:11:49.695302 17542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:11:49.698899 17557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:49.700245 17542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:11:49.700441 17542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f79ba94694cb4b82ad855dcc057c8c05"
format_stamp: "Formatted at 2026-06-19 13:11:49 on dist-test-slave-q11v"
server_key: "11cddab673ae70344552bd2e0ba7c81d"
server_key_iv: "7ae60f4ed3eb41c91e7bd739999fd1de"
server_key_version: "encryptionkey@0"
I20260619 13:11:49.700649 17542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:49.726169 17542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:49.730472 17542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:49.730774 17542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:49.750136 17542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37453
I20260619 13:11:49.750221 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37453 every 8 connection(s)
I20260619 13:11:49.751593 17542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:11:49.755992 17609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:49.760074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17542
I20260619 13:11:49.760351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:11:49.761341 388 external_mini_cluster.cc:1428] Setting key 3be7f09c59845a1e6f789704218de237
I20260619 13:11:49.770463 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: Bootstrap starting.
I20260619 13:11:49.774467 17609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:49.775748 17609 log.cc:824] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:11:49 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:49.779469 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: No bootstrap required, opened a new log
I20260619 13:11:49.785885 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:49.763799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.97.62:37453)
0619 13:11:49.764601 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:49.764613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:49.764652 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:11:49.766462 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:49.766486 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:49.766908 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:49.767321 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:49.768795 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:49.770206 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:49.771610 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:49.772332 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:49.772529 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:49.777462 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:49.777470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:49.777486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:49.777527 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:49.780929 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:49.781991 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:49.781998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:49.782004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:49.782177 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:49.782724 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:49.782729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:49.782731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:49.783364 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:49.783628 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:49.784158 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 13:11:49.784531 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260619 13:11:49.790704 17609 raft_consensus.cc:348] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } }
I20260619 13:11:49.791154 17609 raft_consensus.cc:374] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:49.791287 17609 raft_consensus.cc:729] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ba94694cb4b82ad855dcc057c8c05, State: Initialized, Role: FOLLOWER
I20260619 13:11:49.791927 17609 consensus_queue.cc:260] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } }
I20260619 13:11:49.792227 17609 raft_consensus.cc:388] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:11:49.792397 17609 raft_consensus.cc:482] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:11:49.792613 17609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:49.795421 17609 raft_consensus.cc:504] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } }
I20260619 13:11:49.795993 17609 leader_election.cc:302] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f79ba94694cb4b82ad855dcc057c8c05; no voters:
I20260619 13:11:49.796662 17609 leader_election.cc:288] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:11:49.796775 17614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:49.797737 17614 raft_consensus.cc:686] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [term 1 LEADER]: Becoming Leader. State: Replica: f79ba94694cb4b82ad855dcc057c8c05, State: Running, Role: LEADER
I20260619 13:11:49.798319 17614 consensus_queue.cc:237] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } }
I20260619 13:11:49.799291 17609 sys_catalog.cc:564] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:11:49.801011 17615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f79ba94694cb4b82ad855dcc057c8c05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } } }
I20260619 13:11:49.801090 17616 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f79ba94694cb4b82ad855dcc057c8c05. Latest consensus state: current_term: 1 leader_uuid: "f79ba94694cb4b82ad855dcc057c8c05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ba94694cb4b82ad855dcc057c8c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37453 } } }
I20260619 13:11:49.801688 17616 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:49.802410 17615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:49.802855 17620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:11:49.809979 17620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:11:49.820191 17620 catalog_manager.cc:1347] Generated new cluster ID: 379fc823de4c4b17b377e45063c6ce65
I20260619 13:11:49.820320 17620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:11:49.828310 17620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:11:49.829700 17620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:11:49.836052 17620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: Generated new TSK 0
I20260619 13:11:49.836833 17620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:11:49.903573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:50.249133 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:50.249462 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:50.249686 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:50.256875 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:50.257035 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:50.257222 17635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:11:50.267053 17635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:50.268378 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:50.269861 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:50.283896 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:50.283941 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:50.283892 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:50.284827 17635 server_base.cc:1034] running on GCE node
I20260619 13:11:50.285490 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:50.286441 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:50.287667 17635 hybrid_clock.cc:648] HybridClock initialized: now 1781874710287632 us; error 49 us; skew 500 ppm
I20260619 13:11:50.287983 17635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:50 dist-test-slave-q11v krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:50.291139 17635 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:11:50.292519 17635 webserver.cc:460] Webserver started at http://127.0.97.1:38765/ using document root <none> and password file <none>
I20260619 13:11:50.293201 17635 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:50.293347 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:50.293723 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:50.296563 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "e7ea48d36a2344828be124c40c9cc547"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "58dae71820c9cf62db4e36647198c262"
server_key_iv: "93b1baa73911bc4b7abd3dfcef60ad4b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.297353 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e7ea48d36a2344828be124c40c9cc547"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "58dae71820c9cf62db4e36647198c262"
server_key_iv: "93b1baa73911bc4b7abd3dfcef60ad4b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.302561 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:11:50.306339 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.307695 17635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:11:50.307885 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "e7ea48d36a2344828be124c40c9cc547"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "58dae71820c9cf62db4e36647198c262"
server_key_iv: "93b1baa73911bc4b7abd3dfcef60ad4b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.308107 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:50.327457 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:50.331475 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:50.331769 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:50.332722 17635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:50.334177 17635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:50.334278 17635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.334429 17635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:50.334501 17635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.367183 17635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42155
I20260619 13:11:50.367229 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42155 every 8 connection(s)
I20260619 13:11:50.368472 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:11:50.370987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17635
I20260619 13:11:50.371187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:11:50.371933 388 external_mini_cluster.cc:1428] Setting key 72f0cd320ae3e548f1641c4e5bb2e848
Jun 19 13:11:50 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:50.392467 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:50.371035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35755 (local address 127.0.97.62:37453)
0619 13:11:50.371548 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:50.371555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:50.372689 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 13:11:50.375099 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:50.375103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:50.375165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:50.375256 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:50.377361 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.378136 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.380663 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.381072 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.381181 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:50.385255 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:50.385265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:50.385273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:50.385319 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:50.388944 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:50.389842 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:50.389851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:50.389857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:50.390012 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:50.390429 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:50.390437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:50.390442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:50.390856 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:50.391009 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:50.391719 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 13:11:50.392003 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260619 13:11:50.393208 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:50.371625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37453 (local address 127.0.97.1:35755)
0619 13:11:50.372495 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:50.372550 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:50.374815 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:50.375531 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:50.375546 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:50.376379 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:50.377184 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:50.377212 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.378302 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.378310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:50.380512 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:50.380528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.381317 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.381327 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:50.381670 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:50.382430 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:50.382464 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:50.385022 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:50.389130 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:50.389143 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:50.389162 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:50.389705 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:50.390122 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:50.390126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:50.390128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:50.390308 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:50.390975 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:50.390981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:50.391317 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 13:11:50.391550 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 13:11:50.391960 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":360,"threads_started":1}
I20260619 13:11:50.394994 17763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37453
I20260619 13:11:50.395357 17763 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:50.396147 17763 heartbeater.cc:511] Master 127.0.97.62:37453 requested a full tablet report, sending...
I20260619 13:11:50.398525 17574 ts_manager.cc:194] Registered new tserver with Master: e7ea48d36a2344828be124c40c9cc547 (127.0.97.1:42155)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:11:50.400517 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:11:50.430799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:50.762513 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:50.762863 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:50.762998 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:50.771378 17769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:50.771493 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:50.771711 17769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:11:50.781939 17769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:50.783345 17769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:50.784734 17769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:50.798266 17774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:50.798175 17775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:50.798218 17777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:50.799391 17769 server_base.cc:1034] running on GCE node
I20260619 13:11:50.799989 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:50.800907 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:50.802143 17769 hybrid_clock.cc:648] HybridClock initialized: now 1781874710802099 us; error 43 us; skew 500 ppm
I20260619 13:11:50.802470 17769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:50 dist-test-slave-q11v krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:50.805703 17769 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:11:50.807147 17769 webserver.cc:460] Webserver started at http://127.0.97.2:45347/ using document root <none> and password file <none>
I20260619 13:11:50.807796 17769 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:50.807893 17769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:50.808156 17769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:50.811236 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "fbb9ee76d3d34c248a80f6000a219968"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "da8b54b92665f978a1d4b4dd93116bbf"
server_key_iv: "55b0dce5f0b27f5276f9acfd7142b81b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.812075 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fbb9ee76d3d34c248a80f6000a219968"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "da8b54b92665f978a1d4b4dd93116bbf"
server_key_iv: "55b0dce5f0b27f5276f9acfd7142b81b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.817226 17769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:11:50.820874 17784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.822304 17769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:11:50.822495 17769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "fbb9ee76d3d34c248a80f6000a219968"
format_stamp: "Formatted at 2026-06-19 13:11:50 on dist-test-slave-q11v"
server_key: "da8b54b92665f978a1d4b4dd93116bbf"
server_key_iv: "55b0dce5f0b27f5276f9acfd7142b81b"
server_key_version: "encryptionkey@0"
I20260619 13:11:50.822710 17769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:50.836850 17769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:50.841852 17769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:50.842118 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:50.843051 17769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:50.844391 17769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:50.844491 17769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.844583 17769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:50.844632 17769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:50.876832 17769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40897
I20260619 13:11:50.876870 17896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40897 every 8 connection(s)
I20260619 13:11:50.878271 17769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:11:50.885416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17769
I20260619 13:11:50.885599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:11:50.886096 388 external_mini_cluster.cc:1428] Setting key f0a17e930c4fd3528bfe9ef7b93b4195
Jun 19 13:11:50 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:50.900767 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:50.880972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33275 (local address 127.0.97.62:37453)
0619 13:11:50.881177 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:50.881181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:50.882537 (+ 1356us) server_negotiation.cc:407] Connection header received
0619 13:11:50.884458 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:50.884464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:50.884565 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:50.884686 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:50.886491 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.887342 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.889041 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.889869 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.890020 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:50.893712 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:50.893728 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:50.893736 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:50.893781 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:50.896933 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:50.898071 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:50.898075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:50.898078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:50.898208 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:50.898705 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:50.898708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:50.898710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:50.898983 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:50.899153 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:50.900041 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 13:11:50.900304 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:11:50.902293 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:50.881497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37453 (local address 127.0.97.2:33275)
0619 13:11:50.882279 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:50.882327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:50.884166 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:50.884930 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:50.884940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:50.885509 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:50.886268 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:50.886288 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.887488 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.887493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:50.888890 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:50.888902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:50.890112 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:50.890117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:50.890356 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:50.891147 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:50.891170 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:50.893420 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:50.897211 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:50.897226 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:50.897254 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:50.897950 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:50.898324 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:50.898332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:50.898339 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:50.898593 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:50.899108 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:50.899119 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:50.899612 (+ 493us) client_negotiation.cc:769] Sending connection context
0619 13:11:50.899937 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 13:11:50.900686 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":284,"threads_started":1}
I20260619 13:11:50.903896 17897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37453
I20260619 13:11:50.904214 17897 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:50.905112 17897 heartbeater.cc:511] Master 127.0.97.62:37453 requested a full tablet report, sending...
I20260619 13:11:50.906860 17574 ts_manager.cc:194] Registered new tserver with Master: fbb9ee76d3d34c248a80f6000a219968 (127.0.97.2:40897)
I20260619 13:11:50.908115 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33275
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:11:50.945533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:51.280174 17902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:51.280489 17902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:51.280655 17902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:51.288525 17902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:51.288674 17902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:51.288864 17902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:11:51.298807 17902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:51.300168 17902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:51.301520 17902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:51.315320 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:51.315367 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:51.315390 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:51.316246 17902 server_base.cc:1034] running on GCE node
I20260619 13:11:51.316924 17902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:51.317947 17902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:51.319175 17902 hybrid_clock.cc:648] HybridClock initialized: now 1781874711319113 us; error 54 us; skew 500 ppm
I20260619 13:11:51.319504 17902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:51 dist-test-slave-q11v krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:51.322712 17902 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:11:51.324070 17902 webserver.cc:460] Webserver started at http://127.0.97.3:34829/ using document root <none> and password file <none>
I20260619 13:11:51.324733 17902 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:51.324832 17902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:51.325099 17902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:51.328213 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6bf37f9a7c8461b81874ed6b915dd8c"
format_stamp: "Formatted at 2026-06-19 13:11:51 on dist-test-slave-q11v"
server_key: "69037824b7dd81ae7de923d12807a532"
server_key_iv: "10ba81282d45cedd2f9189479853a679"
server_key_version: "encryptionkey@0"
I20260619 13:11:51.328959 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6bf37f9a7c8461b81874ed6b915dd8c"
format_stamp: "Formatted at 2026-06-19 13:11:51 on dist-test-slave-q11v"
server_key: "69037824b7dd81ae7de923d12807a532"
server_key_iv: "10ba81282d45cedd2f9189479853a679"
server_key_version: "encryptionkey@0"
I20260619 13:11:51.334180 17902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:11:51.338042 17917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:51.339488 17902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:11:51.339635 17902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "e6bf37f9a7c8461b81874ed6b915dd8c"
format_stamp: "Formatted at 2026-06-19 13:11:51 on dist-test-slave-q11v"
server_key: "69037824b7dd81ae7de923d12807a532"
server_key_iv: "10ba81282d45cedd2f9189479853a679"
server_key_version: "encryptionkey@0"
I20260619 13:11:51.339843 17902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:51.385660 17902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:51.390326 17902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:51.390594 17902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:51.391594 17902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:51.392952 17902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:51.393028 17902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:51.393119 17902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:51.393168 17902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:51.403966 17763 heartbeater.cc:503] Master 127.0.97.62:37453 was elected leader, sending a full tablet report...
I20260619 13:11:51.428603 17902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42267
I20260619 13:11:51.428647 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42267 every 8 connection(s)
I20260619 13:11:51.430055 17902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:11:51.431021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 17902
I20260619 13:11:51.431200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:11:51.431761 388 external_mini_cluster.cc:1428] Setting key 4329520e9df7ab8457c309fb022d8f18
Jun 19 13:11:51 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:51.451861 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.432964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60929 (local address 127.0.97.62:37453)
0619 13:11:51.433487 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.433493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.434809 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 13:11:51.436856 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.436860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.436928 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.437034 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:51.438647 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.439432 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.440953 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.441349 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.441449 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.444994 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:51.444999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:51.445003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:51.445038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:51.448643 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.449365 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.449369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.449371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.449491 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.449978 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.449982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.449984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.450497 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:51.450669 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.451392 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.451564 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260619 13:11:51.452889 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.433787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37453 (local address 127.0.97.3:60929)
0619 13:11:51.434636 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.434687 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.436569 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.437248 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.437257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.437881 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:51.438462 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.438480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.439567 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.439570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.440802 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.440814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.441546 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.441551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.441846 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.442483 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:51.442505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:51.444787 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:51.448778 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:51.448785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:51.448800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:51.449249 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:51.449579 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:51.449582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:51.449584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:51.449873 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:51.450659 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:51.450666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:51.451014 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.451269 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.451725 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":357,"threads_started":1}
I20260619 13:11:51.454411 18030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37453
I20260619 13:11:51.454746 18030 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:51.455476 18030 heartbeater.cc:511] Master 127.0.97.62:37453 requested a full tablet report, sending...
I20260619 13:11:51.456981 17574 ts_manager.cc:194] Registered new tserver with Master: e6bf37f9a7c8461b81874ed6b915dd8c (127.0.97.3:42267)
I20260619 13:11:51.457916 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60929
I20260619 13:11:51.460677 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:11:51.478286 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.465308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.97.62:37453)
0619 13:11:51.465553 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.465560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.465942 (+ 382us) server_negotiation.cc:407] Connection header received
0619 13:11:51.466082 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.466086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.466154 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.466242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:51.467692 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.468413 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.469866 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.470258 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.470354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.473038 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:51.473043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:51.473047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:51.473084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:51.475803 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.476484 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.476487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.476490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.476616 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.476942 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.476945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.476947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.477262 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:51.477444 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.477878 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.478053 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:11:51.482649 17574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:11:51.489739 17574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:11:51.518249 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.503292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.97.1:42155)
0619 13:11:51.510048 (+ 6756us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.510059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.510093 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:11:51.510209 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.510218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.510517 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.510735 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:51.511771 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.513375 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.515962 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.516924 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.517085 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.517336 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.517722 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.517896 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6557,"thread_start_us":6394,"threads_started":1}
I20260619 13:11:51.518314 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.502633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42155 (local address 127.0.0.1:59242)
0619 13:11:51.503413 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.503465 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.503695 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.511081 (+ 7386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.511089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.511125 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:51.511559 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.511573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.513513 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.513522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.515804 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.515816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.517172 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.517180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.517428 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.517472 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.517664 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.517879 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":9,"spinlock_wait_cycles":4352,"thread_start_us":1323,"threads_started":1}
I20260619 13:11:51.520475 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.502674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40897 (local address 127.0.0.1:39252)
0619 13:11:51.510302 (+ 7628us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.510321 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.510448 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.512805 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.512809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.512841 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:51.513331 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.513348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.515334 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.515339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.517617 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.517632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.519570 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.519577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.519752 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.519785 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.519857 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.519990 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7513,"mutex_wait_us":19,"thread_start_us":8174,"threads_started":1}
I20260619 13:11:51.521091 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.505202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.0.97.3:42267)
0619 13:11:51.511859 (+ 6657us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.511868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.511901 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:11:51.512067 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.512076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.512325 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.513097 (+ 772us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:51.513396 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.514903 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.517820 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.519926 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.520045 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.520276 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.520549 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.520735 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6470,"thread_start_us":6293,"threads_started":1}
I20260619 13:11:51.521375 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.504918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42267 (local address 127.0.0.1:58774)
0619 13:11:51.511124 (+ 6206us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.511142 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.511258 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.512536 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.512542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.512567 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:51.512960 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.512972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.515040 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.515046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.517627 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.517635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.520147 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.520155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.520332 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.520373 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.520479 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.520861 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6074,"mutex_wait_us":104,"thread_start_us":5623,"threads_started":1}
I20260619 13:11:51.521661 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.503762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.0.97.2:40897)
0619 13:11:51.512253 (+ 8491us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.512262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.512296 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:11:51.512371 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.512376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.512603 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.512807 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:51.513585 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.515191 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.517820 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.519343 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.520845 (+ 1502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.521045 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.521115 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.521302 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8288,"thread_start_us":8204,"threads_started":1}
I20260619 13:11:51.523038 17698 tablet_service.cc:1467] Processing CreateTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 (DEFAULT_TABLE table=test-table [id=f5093ad009184f01b2cf66fc2423e379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:51.524659 17698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f08b53e8f14cc891e739c2a1ea30f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:51.524773 17832 tablet_service.cc:1467] Processing CreateTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 (DEFAULT_TABLE table=test-table [id=f5093ad009184f01b2cf66fc2423e379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:51.526271 17832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f08b53e8f14cc891e739c2a1ea30f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:51.526583 17965 tablet_service.cc:1467] Processing CreateTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 (DEFAULT_TABLE table=test-table [id=f5093ad009184f01b2cf66fc2423e379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:51.527985 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f08b53e8f14cc891e739c2a1ea30f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:51.537379 18049 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Bootstrap starting.
I20260619 13:11:51.539865 18050 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Bootstrap starting.
I20260619 13:11:51.541551 18049 tablet_bootstrap.cc:654] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:51.543407 18050 tablet_bootstrap.cc:654] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:51.543473 18049 log.cc:824] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:51.544720 18050 log.cc:824] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:51.547099 18049 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: No bootstrap required, opened a new log
I20260619 13:11:51.545059 18051 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Bootstrap starting.
I20260619 13:11:51.547389 18049 ts_tablet_manager.cc:1397] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 13:11:51.547719 18050 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: No bootstrap required, opened a new log
I20260619 13:11:51.547983 18050 ts_tablet_manager.cc:1397] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:11:51.550156 18051 tablet_bootstrap.cc:654] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:51.551151 18051 log.cc:824] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:51.554253 18051 tablet_bootstrap.cc:492] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: No bootstrap required, opened a new log
I20260619 13:11:51.554536 18051 ts_tablet_manager.cc:1397] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:11:51.558338 18049 raft_consensus.cc:348] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.558601 18050 raft_consensus.cc:348] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.558877 18049 raft_consensus.cc:374] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:51.558977 18049 raft_consensus.cc:729] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb9ee76d3d34c248a80f6000a219968, State: Initialized, Role: FOLLOWER
I20260619 13:11:51.559010 18050 raft_consensus.cc:374] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:51.559099 18050 raft_consensus.cc:729] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ea48d36a2344828be124c40c9cc547, State: Initialized, Role: FOLLOWER
I20260619 13:11:51.559703 18050 consensus_queue.cc:260] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.559713 18049 consensus_queue.cc:260] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.562057 18050 ts_tablet_manager.cc:1428] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 13:11:51.563030 17897 heartbeater.cc:503] Master 127.0.97.62:37453 was elected leader, sending a full tablet report...
I20260619 13:11:51.563246 18051 raft_consensus.cc:348] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.563694 18051 raft_consensus.cc:374] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:51.563796 18051 raft_consensus.cc:729] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bf37f9a7c8461b81874ed6b915dd8c, State: Initialized, Role: FOLLOWER
I20260619 13:11:51.563880 18049 ts_tablet_manager.cc:1428] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260619 13:11:51.564577 18051 consensus_queue.cc:260] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.566318 18051 ts_tablet_manager.cc:1428] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 13:11:51.566861 18030 heartbeater.cc:503] Master 127.0.97.62:37453 was elected leader, sending a full tablet report...
W20260619 13:11:51.621022 17764 tablet.cc:2369] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:11:51.630278 17898 tablet.cc:2369] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:51.666446 18055 raft_consensus.cc:482] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:51.666807 18055 raft_consensus.cc:504] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.668571 18055 leader_election.cc:288] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb9ee76d3d34c248a80f6000a219968 (127.0.97.2:40897), e6bf37f9a7c8461b81874ed6b915dd8c (127.0.97.3:42267)
I20260619 13:11:51.675415 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.669208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37961 (local address 127.0.97.2:40897)
0619 13:11:51.669398 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.669407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.670027 (+ 620us) server_negotiation.cc:407] Connection header received
0619 13:11:51.670280 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.670284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.670345 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.670444 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:51.671009 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.671723 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.673699 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.674469 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.674615 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.674726 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.675014 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.675149 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:11:51.675494 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.669156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54627 (local address 127.0.97.3:42267)
0619 13:11:51.669371 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.669377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.670357 (+ 980us) server_negotiation.cc:407] Connection header received
0619 13:11:51.670559 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.670563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.670620 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.670713 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:51.671282 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.671984 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.673796 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.674560 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.674719 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.674831 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.675110 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.675249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:11:51.675609 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.669142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42267 (local address 127.0.97.1:54627)
0619 13:11:51.670233 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.670247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.670405 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.670751 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.670755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.670784 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:51.671114 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.671122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.672125 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.672132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.673610 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.673646 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.674802 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.674808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.674957 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.674980 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.675083 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.675204 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":42,"thread_start_us":740,"threads_started":1}
I20260619 13:11:51.676008 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.668991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40897 (local address 127.0.97.1:37961)
0619 13:11:51.669898 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:51.669915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:51.670124 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:51.670460 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:51.670464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:51.670488 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:51.670867 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.670875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.671852 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.671859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.673527 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:51.673538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.674712 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.674719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:51.674861 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.674883 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:11:51.674989 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:11:51.675115 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":2,"thread_start_us":505,"threads_started":1}
I20260619 13:11:51.676666 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f08b53e8f14cc891e739c2a1ea30f2" candidate_uuid: "e7ea48d36a2344828be124c40c9cc547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" is_pre_election: true
I20260619 13:11:51.676887 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f08b53e8f14cc891e739c2a1ea30f2" candidate_uuid: "e7ea48d36a2344828be124c40c9cc547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb9ee76d3d34c248a80f6000a219968" is_pre_election: true
I20260619 13:11:51.677248 17985 raft_consensus.cc:2454] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ea48d36a2344828be124c40c9cc547 in term 0.
I20260619 13:11:51.677415 17852 raft_consensus.cc:2454] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ea48d36a2344828be124c40c9cc547 in term 0.
I20260619 13:11:51.677945 17654 leader_election.cc:302] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6bf37f9a7c8461b81874ed6b915dd8c, e7ea48d36a2344828be124c40c9cc547; no voters:
I20260619 13:11:51.678403 18055 raft_consensus.cc:2781] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:11:51.678560 18055 raft_consensus.cc:482] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:11:51.678673 18055 raft_consensus.cc:3037] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:51.680878 18055 raft_consensus.cc:504] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.681531 18055 leader_election.cc:288] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [CANDIDATE]: Term 1 election: Requested vote from peers fbb9ee76d3d34c248a80f6000a219968 (127.0.97.2:40897), e6bf37f9a7c8461b81874ed6b915dd8c (127.0.97.3:42267)
W20260619 13:11:51.681854 18031 tablet.cc:2369] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:51.682144 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f08b53e8f14cc891e739c2a1ea30f2" candidate_uuid: "e7ea48d36a2344828be124c40c9cc547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb9ee76d3d34c248a80f6000a219968"
I20260619 13:11:51.682386 17852 raft_consensus.cc:3037] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:51.683001 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f08b53e8f14cc891e739c2a1ea30f2" candidate_uuid: "e7ea48d36a2344828be124c40c9cc547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c"
I20260619 13:11:51.683220 17985 raft_consensus.cc:3037] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:51.685662 17852 raft_consensus.cc:2454] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ea48d36a2344828be124c40c9cc547 in term 1.
I20260619 13:11:51.686142 17652 leader_election.cc:302] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7ea48d36a2344828be124c40c9cc547, fbb9ee76d3d34c248a80f6000a219968; no voters:
I20260619 13:11:51.686362 17985 raft_consensus.cc:2454] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ea48d36a2344828be124c40c9cc547 in term 1.
I20260619 13:11:51.686506 18055 raft_consensus.cc:2781] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:51.687031 18055 raft_consensus.cc:686] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 1 LEADER]: Becoming Leader. State: Replica: e7ea48d36a2344828be124c40c9cc547, State: Running, Role: LEADER
I20260619 13:11:51.687611 18055 consensus_queue.cc:237] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } }
I20260619 13:11:51.691985 17573 catalog_manager.cc:5495] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ea48d36a2344828be124c40c9cc547 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e7ea48d36a2344828be124c40c9cc547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ea48d36a2344828be124c40c9cc547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 } health_report { overall_health: UNKNOWN } } }
I20260619 13:11:51.752786 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.747283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.97.1:42155)
0619 13:11:51.747491 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.747498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.747595 (+ 97us) server_negotiation.cc:407] Connection header received
0619 13:11:51.747870 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.747875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.747951 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.748105 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:11:51.748483 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.749204 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.750658 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.751045 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.751192 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.751668 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:11:51.752214 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:11:51.752381 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.752434 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.752514 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:11:51.761711 17852 raft_consensus.cc:1261] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 1 FOLLOWER]: Refusing update from remote peer e7ea48d36a2344828be124c40c9cc547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:51.761754 17985 raft_consensus.cc:1261] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 1 FOLLOWER]: Refusing update from remote peer e7ea48d36a2344828be124c40c9cc547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:51.762778 18060 consensus_queue.cc:1035] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb9ee76d3d34c248a80f6000a219968" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:51.763233 18055 consensus_queue.cc:1035] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6bf37f9a7c8461b81874ed6b915dd8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:51.773923 18063 mvcc.cc:205] Tried to move back new op lower bound from 7298558819364589568 to 7298558819076059136. Current Snapshot: MvccSnapshot[applied={T|T < 7298558819364589568}]
I20260619 13:11:51.775889 18064 mvcc.cc:205] Tried to move back new op lower bound from 7298558819364589568 to 7298558819076059136. Current Snapshot: MvccSnapshot[applied={T|T < 7298558819364589568}]
I20260619 13:11:51.779595 18065 mvcc.cc:205] Tried to move back new op lower bound from 7298558819364589568 to 7298558819076059136. Current Snapshot: MvccSnapshot[applied={T|T < 7298558819364589568}]
I20260619 13:11:51.789564 17574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:51.789944 17574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:51.793417 17574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f79ba94694cb4b82ad855dcc057c8c05: Sending DeleteTablet for 3 replicas of tablet c7f08b53e8f14cc891e739c2a1ea30f2
I20260619 13:11:51.794755 17832 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39252
I20260619 13:11:51.794867 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58774
I20260619 13:11:51.794965 17698 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f08b53e8f14cc891e739c2a1ea30f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59242
I20260619 13:11:51.796207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17635
I20260619 13:11:51.804898 18072 tablet_replica.cc:330] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: stopping tablet replica
I20260619 13:11:51.804340 18073 tablet_replica.cc:330] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: stopping tablet replica
I20260619 13:11:51.806010 18074 tablet_replica.cc:330] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: stopping tablet replica
I20260619 13:11:51.806021 18073 raft_consensus.cc:2229] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:51.806406 18074 raft_consensus.cc:2229] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:11:51.806546 18073 raft_consensus.cc:2258] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:51.805399 18072 raft_consensus.cc:2229] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:51.807209 18074 raft_consensus.cc:2258] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:51.807377 18072 raft_consensus.cc:2258] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:51.809456 18072 ts_tablet_manager.cc:1905] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:51.809473 18074 ts_tablet_manager.cc:1905] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:51.811539 18073 ts_tablet_manager.cc:1905] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:51.816767 18072 ts_tablet_manager.cc:1918] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:51.816949 18072 log.cc:1196] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/c7f08b53e8f14cc891e739c2a1ea30f2
I20260619 13:11:51.817498 18072 ts_tablet_manager.cc:1939] T c7f08b53e8f14cc891e739c2a1ea30f2 P fbb9ee76d3d34c248a80f6000a219968: Deleting consensus metadata
I20260619 13:11:51.817507 18074 ts_tablet_manager.cc:1918] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:51.817726 18074 log.cc:1196] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/c7f08b53e8f14cc891e739c2a1ea30f2
I20260619 13:11:51.818248 18074 ts_tablet_manager.cc:1939] T c7f08b53e8f14cc891e739c2a1ea30f2 P e7ea48d36a2344828be124c40c9cc547: Deleting consensus metadata
I20260619 13:11:51.819584 17559 catalog_manager.cc:4841] TS fbb9ee76d3d34c248a80f6000a219968 (127.0.97.2:40897): tablet c7f08b53e8f14cc891e739c2a1ea30f2 (table test-table [id=f5093ad009184f01b2cf66fc2423e379]) successfully deleted
I20260619 13:11:51.819972 17561 catalog_manager.cc:4841] TS e7ea48d36a2344828be124c40c9cc547 (127.0.97.1:42155): tablet c7f08b53e8f14cc891e739c2a1ea30f2 (table test-table [id=f5093ad009184f01b2cf66fc2423e379]) successfully deleted
I20260619 13:11:51.820267 18073 ts_tablet_manager.cc:1918] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:51.820441 18073 log.cc:1196] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/c7f08b53e8f14cc891e739c2a1ea30f2
I20260619 13:11:51.820959 18073 ts_tablet_manager.cc:1939] T c7f08b53e8f14cc891e739c2a1ea30f2 P e6bf37f9a7c8461b81874ed6b915dd8c: Deleting consensus metadata
I20260619 13:11:51.822826 17561 catalog_manager.cc:4841] TS e6bf37f9a7c8461b81874ed6b915dd8c (127.0.97.3:42267): tablet c7f08b53e8f14cc891e739c2a1ea30f2 (table test-table [id=f5093ad009184f01b2cf66fc2423e379]) successfully deleted
Jun 19 13:11:51 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:11:51.830974 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.798078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.0.97.1:42155)
0619 13:11:51.798290 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.798296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.803813 (+ 5517us) server_negotiation.cc:407] Connection header received
0619 13:11:51.804454 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.804463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.804584 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.804667 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:51.806705 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.807930 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.813022 (+ 5092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.813792 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.813901 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.824405 (+ 10504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:51.824417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:51.824430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:51.824474 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:51.827933 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.828700 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.828707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.828710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.828860 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.829292 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.829297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.829299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.829723 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:51.830039 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.830374 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.830622 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:11:51.831664 17758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:51.931561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17635
I20260619 13:11:51.949958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17769
Jun 19 13:11:51 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:11:51.966197 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:51.950742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.0.97.2:40897)
0619 13:11:51.950937 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:51.950942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:51.950959 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:11:51.951115 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:51.951119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:51.951195 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:51.951279 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:51.952487 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.953216 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.954537 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:51.954988 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:51.955131 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:51.960162 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:51.960178 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:51.960194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:51.960229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:51.963360 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.964059 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.964066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.964068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.964192 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:51.964573 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:51.964579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:51.964582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:51.964975 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:51.965282 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:51.965578 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:11:51.965841 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:11:51.966830 17892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:52.056939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17769
I20260619 13:11:52.074934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17902
Jun 19 13:11:52 dist-test-slave-q11v krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:11:52.089784 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:52.075735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.0.97.3:42267)
0619 13:11:52.075922 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:52.075927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:52.075945 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:11:52.076083 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:52.076087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:52.076153 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:52.076250 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:52.077685 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:52.078405 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:52.079691 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:52.080074 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:52.080170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:52.084415 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:52.084430 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:52.084447 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:52.084488 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:52.087122 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:52.087781 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:52.087788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:52.087790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:52.087917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:52.088367 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:52.088370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:52.088372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:52.088700 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:52.088916 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:52.089236 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 13:11:52.089397 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:11:52.090404 18025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:52.183627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17902
I20260619 13:11:52.200577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 17542
I20260619 13:11:52.201813 17604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:52.316198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 17542
2026-06-19T13:11:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4396 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T13:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:11:52Z Disabled control of system clock
I20260619 13:11:52.391844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43869
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:11:52.720976 18083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:52.721272 18083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:52.721413 18083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:52.728870 18083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:11:52.728981 18083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:52.729050 18083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:52.729140 18083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:11:52.729195 18083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:11:52.739183 18083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:52.740603 18083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:52.742463 18083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:52.754284 18088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:52.754211 18089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:52.754377 18091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:52.754909 18083 server_base.cc:1034] running on GCE node
I20260619 13:11:52.755630 18083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:52.757035 18083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:52.758262 18083 hybrid_clock.cc:648] HybridClock initialized: now 1781874712758220 us; error 31 us; skew 500 ppm
I20260619 13:11:52.758627 18083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:11:52.764134 18083 webserver.cc:460] Webserver started at http://127.0.97.62:32799/ using document root <none> and password file <none>
I20260619 13:11:52.764904 18083 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:52.765018 18083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:52.765372 18083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:52.768375 18083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "fdb16b259a9e4eb6b631cb139dab2ee3"
format_stamp: "Formatted at 2026-06-19 13:11:52 on dist-test-slave-q11v"
server_key: "b57d045df1803f7c27e5bbae4c41136e"
server_key_iv: "6ae9f116e3f768a38304596f6d3e837d"
server_key_version: "encryptionkey@0"
I20260619 13:11:52.769184 18083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "fdb16b259a9e4eb6b631cb139dab2ee3"
format_stamp: "Formatted at 2026-06-19 13:11:52 on dist-test-slave-q11v"
server_key: "b57d045df1803f7c27e5bbae4c41136e"
server_key_iv: "6ae9f116e3f768a38304596f6d3e837d"
server_key_version: "encryptionkey@0"
I20260619 13:11:52.774423 18083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:11:52.777899 18097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:52.779394 18083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:11:52.779600 18083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "fdb16b259a9e4eb6b631cb139dab2ee3"
format_stamp: "Formatted at 2026-06-19 13:11:52 on dist-test-slave-q11v"
server_key: "b57d045df1803f7c27e5bbae4c41136e"
server_key_iv: "6ae9f116e3f768a38304596f6d3e837d"
server_key_version: "encryptionkey@0"
I20260619 13:11:52.779851 18083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:52.795485 18083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:52.796365 18083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:52.796667 18083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:52.814450 18083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43869
I20260619 13:11:52.814471 18148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43869 every 8 connection(s)
I20260619 13:11:52.815845 18083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:11:52.819588 18149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:52.825875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18083
I20260619 13:11:52.826063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:11:52.826539 388 external_mini_cluster.cc:1428] Setting key 9f572e77dbaa15560dcf9184666b3944
I20260619 13:11:52.830246 18149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: Bootstrap starting.
I20260619 13:11:52.833475 18149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:52.834638 18149 log.cc:824] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:52.837711 18149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: No bootstrap required, opened a new log
I20260619 13:11:52.840559 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:52.828558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.97.62:43869)
0619 13:11:52.829566 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:52.829580 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:52.829686 (+ 106us) server_negotiation.cc:407] Connection header received
0619 13:11:52.831486 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:52.831511 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:52.831973 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:52.832287 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:52.832884 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:52.834249 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:52.835986 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:52.836664 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:52.836801 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:52.838104 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:52.838112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:52.838129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:52.838669 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:52.838744 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:52.838764 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:52.838855 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:52.839128 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:11:52.839495 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":359,"threads_started":1}
I20260619 13:11:52.845556 18149 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } }
I20260619 13:11:52.845984 18149 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:52.846100 18149 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb16b259a9e4eb6b631cb139dab2ee3, State: Initialized, Role: FOLLOWER
I20260619 13:11:52.846714 18149 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } }
I20260619 13:11:52.846941 18149 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:11:52.847069 18149 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:11:52.847234 18149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:52.849732 18149 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } }
I20260619 13:11:52.850286 18149 leader_election.cc:302] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb16b259a9e4eb6b631cb139dab2ee3; no voters:
I20260619 13:11:52.850832 18149 leader_election.cc:288] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:11:52.850982 18154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:52.851719 18154 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [term 1 LEADER]: Becoming Leader. State: Replica: fdb16b259a9e4eb6b631cb139dab2ee3, State: Running, Role: LEADER
I20260619 13:11:52.852253 18154 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } }
I20260619 13:11:52.852895 18149 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:11:52.855116 18155 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } } }
I20260619 13:11:52.855484 18155 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:52.856237 18156 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb16b259a9e4eb6b631cb139dab2ee3. Latest consensus state: current_term: 1 leader_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb16b259a9e4eb6b631cb139dab2ee3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43869 } } }
I20260619 13:11:52.856561 18156 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:52.857725 18161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:11:52.862953 18161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:11:52.871965 18161 catalog_manager.cc:1347] Generated new cluster ID: 00588a2792ce4010a6ed5ea14d41d71e
I20260619 13:11:52.872113 18161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:11:52.901109 18161 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:11:52.903035 18161 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:11:52.908854 18161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: Generated new TSK 0
I20260619 13:11:52.909758 18161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:11:52.916494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--builtin_ntp_servers=127.0.97.20:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:11:53.253698 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:53.254025 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:53.254175 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:53.261535 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:53.261674 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:53.261878 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:11:53.271023 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:53.272253 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:53.274071 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:53.287017 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:53.287039 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:53.287719 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:53.289028 18173 server_base.cc:1034] running on GCE node
I20260619 13:11:53.289731 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:53.290712 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:53.291915 18173 hybrid_clock.cc:648] HybridClock initialized: now 1781874713291848 us; error 56 us; skew 500 ppm
I20260619 13:11:53.292246 18173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:11:53.294569 18173 webserver.cc:460] Webserver started at http://127.0.97.1:45391/ using document root <none> and password file <none>
I20260619 13:11:53.295279 18173 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:53.295393 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:53.295789 18173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:53.299422 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "fcf93444b3394a0cbe4584a810301411"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "ca5fb288ec2e1dde3e9db8fa224d39e7"
server_key_iv: "278da2719983fcbe7353ae0855089212"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.300271 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fcf93444b3394a0cbe4584a810301411"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "ca5fb288ec2e1dde3e9db8fa224d39e7"
server_key_iv: "278da2719983fcbe7353ae0855089212"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.305439 18173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:11:53.308941 18187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.310369 18173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:11:53.310542 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "fcf93444b3394a0cbe4584a810301411"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "ca5fb288ec2e1dde3e9db8fa224d39e7"
server_key_iv: "278da2719983fcbe7353ae0855089212"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.310781 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:53.334164 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:53.335059 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:53.335381 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:53.336402 18173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:53.337963 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:53.338089 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.338223 18173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:53.338313 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.372430 18173 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33013
I20260619 13:11:53.372520 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33013 every 8 connection(s)
I20260619 13:11:53.373813 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:11:53.374755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18173
I20260619 13:11:53.374933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:11:53.375494 388 external_mini_cluster.cc:1428] Setting key e07598a2c60437f414b792d0086713cd
I20260619 13:11:53.378794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--builtin_ntp_servers=127.0.97.20:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:11:53.387480 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:53.376727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46467 (local address 127.0.97.62:43869)
0619 13:11:53.377577 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:53.377583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:53.378720 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 13:11:53.380630 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:53.380636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:53.380741 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:53.380859 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:53.382072 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.382818 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.384503 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.385014 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.385157 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:53.386621 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:53.386631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:53.386635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:53.386654 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:53.386695 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:53.386706 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:53.386799 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:53.387036 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 13:11:53.387141 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":607,"threads_started":1}
I20260619 13:11:53.388473 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:53.377536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43869 (local address 127.0.97.1:46467)
0619 13:11:53.378547 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:53.378610 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:53.380349 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:53.381108 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:53.381116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:53.381217 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:53.381897 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:53.381917 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.382979 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.382983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:53.384360 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:53.384373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.385259 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.385267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:53.385513 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:53.386296 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:11:53.386299 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:53.386340 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:11:53.386425 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:53.386813 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:53.386828 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:53.386840 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 13:11:53.387011 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 13:11:53.387414 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":311,"threads_started":1}
I20260619 13:11:53.390232 18300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43869
I20260619 13:11:53.390573 18300 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:53.391358 18300 heartbeater.cc:511] Master 127.0.97.62:43869 requested a full tablet report, sending...
I20260619 13:11:53.393750 18114 ts_manager.cc:194] Registered new tserver with Master: fcf93444b3394a0cbe4584a810301411 (127.0.97.1:33013)
I20260619 13:11:53.396128 18114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46467
W20260619 13:11:53.707664 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:53.707973 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:53.708062 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:53.715371 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:53.715483 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:53.715669 18304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:11:53.725013 18304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:53.726406 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:53.728271 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:53.741721 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:53.741647 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:53.741690 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:53.742283 18304 server_base.cc:1034] running on GCE node
I20260619 13:11:53.742980 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:53.743940 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:53.745136 18304 hybrid_clock.cc:648] HybridClock initialized: now 1781874713745077 us; error 57 us; skew 500 ppm
I20260619 13:11:53.745450 18304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:11:53.747743 18304 webserver.cc:460] Webserver started at http://127.0.97.2:39949/ using document root <none> and password file <none>
I20260619 13:11:53.748425 18304 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:53.748538 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:53.748868 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:53.751912 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "b23f6fa8e7f64137b437b99e0477c946"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "8ef2ad1760904718d50a8939c1a9c518"
server_key_iv: "1e0ac6ea027d7d48ded60845d350d0fd"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.752735 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b23f6fa8e7f64137b437b99e0477c946"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "8ef2ad1760904718d50a8939c1a9c518"
server_key_iv: "1e0ac6ea027d7d48ded60845d350d0fd"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.758419 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:11:53.762032 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.763414 18304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:11:53.763576 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "b23f6fa8e7f64137b437b99e0477c946"
format_stamp: "Formatted at 2026-06-19 13:11:53 on dist-test-slave-q11v"
server_key: "8ef2ad1760904718d50a8939c1a9c518"
server_key_iv: "1e0ac6ea027d7d48ded60845d350d0fd"
server_key_version: "encryptionkey@0"
I20260619 13:11:53.763804 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:53.790503 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:53.791400 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:53.791697 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:53.792793 18304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:53.794391 18304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:53.794507 18304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.794656 18304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:53.794755 18304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:53.829555 18304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38869
I20260619 13:11:53.829630 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38869 every 8 connection(s)
I20260619 13:11:53.830941 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:11:53.835536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18304
I20260619 13:11:53.835736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:11:53.836277 388 external_mini_cluster.cc:1428] Setting key a4d8873d4aba6d32ff20a313eb83ef32
I20260619 13:11:53.844069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--builtin_ntp_servers=127.0.97.20:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:11:53.845853 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:53.833704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52571 (local address 127.0.97.62:43869)
0619 13:11:53.833892 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:53.833896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:53.835399 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 13:11:53.838282 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:53.838286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:53.838355 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:53.838445 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:53.840054 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.840762 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.842869 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.843420 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.843569 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:53.844908 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:53.844917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:53.844925 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:53.844950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:53.844995 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:53.845013 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:53.845113 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:53.845319 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 13:11:53.845431 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:11:53.846807 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:53.834188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43869 (local address 127.0.97.2:52571)
0619 13:11:53.835190 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:53.835251 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:53.837928 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:53.838763 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:53.838777 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:53.838920 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:53.839857 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:53.839884 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.840910 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.840917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:53.842705 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:53.842720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:53.843631 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:53.843636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:53.843850 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:53.844574 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:11:53.844577 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:53.844616 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:11:53.844700 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:53.845109 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:53.845117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:53.845122 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:11:53.845265 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 13:11:53.845565 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":290,"threads_started":1}
I20260619 13:11:53.848199 18431 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43869
I20260619 13:11:53.848528 18431 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:53.849455 18431 heartbeater.cc:511] Master 127.0.97.62:43869 requested a full tablet report, sending...
I20260619 13:11:53.851037 18114 ts_manager.cc:194] Registered new tserver with Master: b23f6fa8e7f64137b437b99e0477c946 (127.0.97.2:38869)
I20260619 13:11:53.851852 18114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52571
W20260619 13:11:54.179394 18434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:54.179698 18434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:54.179785 18434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:54.187331 18434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:54.187444 18434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:54.187552 18434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:11:54.196676 18434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:54.198027 18434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:54.199885 18434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:54.213491 18439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:54.213570 18440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:54.213506 18442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:54.214815 18434 server_base.cc:1034] running on GCE node
I20260619 13:11:54.215509 18434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:54.216612 18434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:54.217844 18434 hybrid_clock.cc:648] HybridClock initialized: now 1781874714217794 us; error 47 us; skew 500 ppm
I20260619 13:11:54.218182 18434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:11:54.220544 18434 webserver.cc:460] Webserver started at http://127.0.97.3:46555/ using document root <none> and password file <none>
I20260619 13:11:54.221279 18434 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:54.221414 18434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:54.221819 18434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:54.224830 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "88a69e1ca4da47f28656db018e035c7f"
format_stamp: "Formatted at 2026-06-19 13:11:54 on dist-test-slave-q11v"
server_key: "0a7046e61a72546e6d3ce71f7828693d"
server_key_iv: "1cf27004528d4adc62890cb76273edbe"
server_key_version: "encryptionkey@0"
I20260619 13:11:54.225728 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "88a69e1ca4da47f28656db018e035c7f"
format_stamp: "Formatted at 2026-06-19 13:11:54 on dist-test-slave-q11v"
server_key: "0a7046e61a72546e6d3ce71f7828693d"
server_key_iv: "1cf27004528d4adc62890cb76273edbe"
server_key_version: "encryptionkey@0"
I20260619 13:11:54.231061 18434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:11:54.234735 18448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:54.236176 18434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:11:54.236366 18434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "88a69e1ca4da47f28656db018e035c7f"
format_stamp: "Formatted at 2026-06-19 13:11:54 on dist-test-slave-q11v"
server_key: "0a7046e61a72546e6d3ce71f7828693d"
server_key_iv: "1cf27004528d4adc62890cb76273edbe"
server_key_version: "encryptionkey@0"
I20260619 13:11:54.236604 18434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:54.254746 18434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:54.255647 18434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:54.255949 18434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:54.256965 18434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:54.258605 18434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:54.258728 18434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:54.258860 18434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:54.258944 18434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:54.294736 18434 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37131
I20260619 13:11:54.294796 18560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37131 every 8 connection(s)
I20260619 13:11:54.296213 18434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:11:54.300231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18434
I20260619 13:11:54.300423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:11:54.300895 388 external_mini_cluster.cc:1428] Setting key 205a6ccc30587e444716cd3552024317
I20260619 13:11:54.309141 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.298878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35609 (local address 127.0.97.62:43869)
0619 13:11:54.299059 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.299064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.300608 (+ 1544us) server_negotiation.cc:407] Connection header received
0619 13:11:54.302611 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.302616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.302708 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.302824 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:54.303909 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.304657 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.306337 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.306754 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.306894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.308319 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:54.308324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:54.308328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:54.308348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:54.308380 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:54.308392 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:54.308490 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.308712 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.308813 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:11:54.310290 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.299385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43869 (local address 127.0.97.3:35609)
0619 13:11:54.300380 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.300432 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.302333 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.303076 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.303084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.303182 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:54.303747 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.303764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.304819 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.304824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.306187 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.306199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.306981 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.306989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.307234 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.307972 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:11:54.307975 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:54.308017 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:11:54.308107 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:54.308488 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:54.308500 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:54.308510 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.308703 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.309130 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":284,"threads_started":1}
I20260619 13:11:54.311695 18561 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43869
I20260619 13:11:54.312050 18561 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:54.312847 18561 heartbeater.cc:511] Master 127.0.97.62:43869 requested a full tablet report, sending...
I20260619 13:11:54.314399 18114 ts_manager.cc:194] Registered new tserver with Master: 88a69e1ca4da47f28656db018e035c7f (127.0.97.3:37131)
I20260619 13:11:54.315550 18114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35609
I20260619 13:11:54.316222 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:11:54.326228 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.320498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.0.97.62:43869)
0619 13:11:54.320702 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.320707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.321022 (+ 315us) server_negotiation.cc:407] Connection header received
0619 13:11:54.321203 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.321208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.321287 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.321381 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:54.321955 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.322688 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.324084 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.324482 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.324598 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.325559 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:54.325563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:54.325587 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:54.325664 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:54.325706 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:54.325720 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:54.325792 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.325916 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.325992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:11:54.330287 18114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:11:54.337074 18114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:11:54.359774 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.350003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.0.97.1:33013)
0619 13:11:54.351839 (+ 1836us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.351850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.351886 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:11:54.352009 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.352017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.352286 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.352504 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:54.353271 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.354943 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.356903 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.358405 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.358540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.358782 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.359207 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.359416 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":1542,"threads_started":1}
I20260619 13:11:54.359778 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.349411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:58972)
0619 13:11:54.351141 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.351164 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.351297 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.352505 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.352512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.352540 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:54.353015 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.353028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.355079 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.355086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.356685 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.356698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.358609 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.358617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.358856 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.358963 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.359126 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.359349 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"mutex_wait_us":3,"thread_start_us":1470,"threads_started":1}
I20260619 13:11:54.361649 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.349365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38869 (local address 127.0.0.1:57446)
0619 13:11:54.350243 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.350292 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.350505 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.353959 (+ 3454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.353964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.353992 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:54.354357 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.354369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.356456 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.356462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.358998 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.359012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.360898 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.360904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.361081 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.361141 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.361214 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.361342 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":101,"thread_start_us":437,"threads_started":1}
I20260619 13:11:54.362573 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.349482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.0.97.2:38869)
0619 13:11:54.353193 (+ 3711us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.353204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.353234 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:11:54.353327 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.353334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.353703 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.354535 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:54.354766 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.356316 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.359291 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.360680 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.361905 (+ 1225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.362153 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.362205 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.362327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3544,"thread_start_us":3427,"threads_started":1}
I20260619 13:11:54.363082 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.350776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37131 (local address 127.0.0.1:33084)
0619 13:11:54.353776 (+ 3000us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.353797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.353911 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.354549 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.354553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.354576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:54.354972 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.354981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.356827 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.356833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.359126 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.359140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.362310 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.362316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.362484 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.362515 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.362642 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.362763 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869,"mutex_wait_us":36,"thread_start_us":2711,"threads_started":1}
I20260619 13:11:54.363384 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.351459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.97.3:37131)
0619 13:11:54.352285 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.352293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.354030 (+ 1737us) server_negotiation.cc:407] Connection header received
0619 13:11:54.354138 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.354146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.354361 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.354540 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:54.355175 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.356656 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.359344 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.361788 (+ 2444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.361926 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.362190 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.362724 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.362890 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":493,"threads_started":1}
I20260619 13:11:54.363987 18235 tablet_service.cc:1467] Processing CreateTablet for tablet 0c319194853545ccba8734fc7d23f1cf (DEFAULT_TABLE table=test-table [id=cf04357189884a4096b30d2a87fb4dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:54.365443 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c319194853545ccba8734fc7d23f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:54.365991 18366 tablet_service.cc:1467] Processing CreateTablet for tablet 0c319194853545ccba8734fc7d23f1cf (DEFAULT_TABLE table=test-table [id=cf04357189884a4096b30d2a87fb4dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:54.367233 18366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c319194853545ccba8734fc7d23f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:54.367141 18496 tablet_service.cc:1467] Processing CreateTablet for tablet 0c319194853545ccba8734fc7d23f1cf (DEFAULT_TABLE table=test-table [id=cf04357189884a4096b30d2a87fb4dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:54.368494 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c319194853545ccba8734fc7d23f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:54.387518 18580 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Bootstrap starting.
I20260619 13:11:54.388607 18579 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Bootstrap starting.
I20260619 13:11:54.392511 18581 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Bootstrap starting.
I20260619 13:11:54.393196 18579 tablet_bootstrap.cc:654] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:54.394068 18580 tablet_bootstrap.cc:654] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:54.394441 18579 log.cc:824] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:54.395344 18580 log.cc:824] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:54.396548 18581 tablet_bootstrap.cc:654] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:54.397527 18579 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: No bootstrap required, opened a new log
I20260619 13:11:54.397847 18579 ts_tablet_manager.cc:1397] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:11:54.398073 18581 log.cc:824] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:54.399528 18300 heartbeater.cc:503] Master 127.0.97.62:43869 was elected leader, sending a full tablet report...
I20260619 13:11:54.401575 18581 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: No bootstrap required, opened a new log
I20260619 13:11:54.401942 18581 ts_tablet_manager.cc:1397] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 13:11:54.403129 18580 tablet_bootstrap.cc:492] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: No bootstrap required, opened a new log
I20260619 13:11:54.403473 18580 ts_tablet_manager.cc:1397] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260619 13:11:54.409178 18579 raft_consensus.cc:348] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.409678 18579 raft_consensus.cc:374] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:54.409839 18579 raft_consensus.cc:729] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf93444b3394a0cbe4584a810301411, State: Initialized, Role: FOLLOWER
I20260619 13:11:54.410502 18579 consensus_queue.cc:260] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.412369 18579 ts_tablet_manager.cc:1428] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 13:11:54.411424 18580 raft_consensus.cc:348] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.412712 18580 raft_consensus.cc:374] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:54.412827 18580 raft_consensus.cc:729] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f6fa8e7f64137b437b99e0477c946, State: Initialized, Role: FOLLOWER
I20260619 13:11:54.412678 18581 raft_consensus.cc:348] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.413148 18581 raft_consensus.cc:374] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:54.413269 18581 raft_consensus.cc:729] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a69e1ca4da47f28656db018e035c7f, State: Initialized, Role: FOLLOWER
I20260619 13:11:54.413560 18580 consensus_queue.cc:260] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.413885 18581 consensus_queue.cc:260] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.414940 18431 heartbeater.cc:503] Master 127.0.97.62:43869 was elected leader, sending a full tablet report...
I20260619 13:11:54.415601 18580 ts_tablet_manager.cc:1428] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260619 13:11:54.415791 18561 heartbeater.cc:503] Master 127.0.97.62:43869 was elected leader, sending a full tablet report...
I20260619 13:11:54.417073 18581 ts_tablet_manager.cc:1428] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
W20260619 13:11:54.548023 18562 tablet.cc:2369] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:11:54.583166 18432 tablet.cc:2369] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:54.611322 18587 raft_consensus.cc:482] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:54.611608 18587 raft_consensus.cc:504] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.613408 18587 leader_election.cc:288] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23f6fa8e7f64137b437b99e0477c946 (127.0.97.2:38869), fcf93444b3394a0cbe4584a810301411 (127.0.97.1:33013)
I20260619 13:11:54.619361 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.613832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49453 (local address 127.0.97.2:38869)
0619 13:11:54.614058 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.614066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.614201 (+ 135us) server_negotiation.cc:407] Connection header received
0619 13:11:54.614434 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.614440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.614555 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.614647 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:54.615101 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.615905 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.617628 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.618465 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.618599 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.618688 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.618944 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.619073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:11:54.619426 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.613766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38869 (local address 127.0.97.3:49453)
0619 13:11:54.614069 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.614089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.614281 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.614709 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.614714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.614772 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:54.614952 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.614961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.616048 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.616053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.617452 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.617463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.618622 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.618627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.618789 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.618812 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.618919 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.619053 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":43392}
I20260619 13:11:54.619860 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.614023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39533 (local address 127.0.97.1:33013)
0619 13:11:54.614237 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.614244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.614688 (+ 444us) server_negotiation.cc:407] Connection header received
0619 13:11:54.614903 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.614908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.614976 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.615077 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:54.615556 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.616227 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.618028 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.618922 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.619060 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.619167 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.619492 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.619625 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 13:11:54.620374 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.613790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.97.3:39533)
0619 13:11:54.614574 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.614588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:54.614759 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:54.615091 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:54.615095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:54.615118 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:54.615403 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.615411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.616391 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.616396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.617859 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:54.617870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.619127 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.619135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:54.619306 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.619327 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:11:54.620057 (+ 730us) client_negotiation.cc:240] Negotiation successful
0619 13:11:54.620145 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":442,"threads_started":1}
I20260619 13:11:54.620426 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c319194853545ccba8734fc7d23f1cf" candidate_uuid: "88a69e1ca4da47f28656db018e035c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f6fa8e7f64137b437b99e0477c946" is_pre_election: true
I20260619 13:11:54.621070 18386 raft_consensus.cc:2454] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a69e1ca4da47f28656db018e035c7f in term 0.
I20260619 13:11:54.621428 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c319194853545ccba8734fc7d23f1cf" candidate_uuid: "88a69e1ca4da47f28656db018e035c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf93444b3394a0cbe4584a810301411" is_pre_election: true
I20260619 13:11:54.621821 18449 leader_election.cc:302] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a69e1ca4da47f28656db018e035c7f, b23f6fa8e7f64137b437b99e0477c946; no voters:
I20260619 13:11:54.621927 18255 raft_consensus.cc:2454] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a69e1ca4da47f28656db018e035c7f in term 0.
I20260619 13:11:54.622225 18587 raft_consensus.cc:2781] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:11:54.622378 18587 raft_consensus.cc:482] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:11:54.622493 18587 raft_consensus.cc:3037] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:54.624667 18587 raft_consensus.cc:504] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.625211 18587 leader_election.cc:288] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [CANDIDATE]: Term 1 election: Requested vote from peers b23f6fa8e7f64137b437b99e0477c946 (127.0.97.2:38869), fcf93444b3394a0cbe4584a810301411 (127.0.97.1:33013)
I20260619 13:11:54.625648 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c319194853545ccba8734fc7d23f1cf" candidate_uuid: "88a69e1ca4da47f28656db018e035c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f6fa8e7f64137b437b99e0477c946"
I20260619 13:11:54.625654 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c319194853545ccba8734fc7d23f1cf" candidate_uuid: "88a69e1ca4da47f28656db018e035c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf93444b3394a0cbe4584a810301411"
I20260619 13:11:54.625850 18386 raft_consensus.cc:3037] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:54.625850 18255 raft_consensus.cc:3037] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 0 FOLLOWER]: Advancing to term 1
W20260619 13:11:54.626263 18301 tablet.cc:2369] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:54.629115 18255 raft_consensus.cc:2454] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a69e1ca4da47f28656db018e035c7f in term 1.
I20260619 13:11:54.629117 18386 raft_consensus.cc:2454] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a69e1ca4da47f28656db018e035c7f in term 1.
I20260619 13:11:54.629541 18451 leader_election.cc:302] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a69e1ca4da47f28656db018e035c7f, fcf93444b3394a0cbe4584a810301411; no voters:
I20260619 13:11:54.629879 18587 raft_consensus.cc:2781] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:54.630324 18587 raft_consensus.cc:686] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 1 LEADER]: Becoming Leader. State: Replica: 88a69e1ca4da47f28656db018e035c7f, State: Running, Role: LEADER
I20260619 13:11:54.630792 18587 consensus_queue.cc:237] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } }
I20260619 13:11:54.635566 18113 catalog_manager.cc:5495] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a69e1ca4da47f28656db018e035c7f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "88a69e1ca4da47f28656db018e035c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a69e1ca4da47f28656db018e035c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260619 13:11:54.670614 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.664725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.97.3:37131)
0619 13:11:54.664960 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.664967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.665077 (+ 110us) server_negotiation.cc:407] Connection header received
0619 13:11:54.665342 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.665346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.665434 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.665594 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:11:54.665988 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.666751 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.668308 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.668769 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.668915 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.669447 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:11:54.670004 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:11:54.670174 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.670257 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.670336 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:11:54.678977 18255 raft_consensus.cc:1261] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 1 FOLLOWER]: Refusing update from remote peer 88a69e1ca4da47f28656db018e035c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:54.678974 18386 raft_consensus.cc:1261] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 1 FOLLOWER]: Refusing update from remote peer 88a69e1ca4da47f28656db018e035c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:54.680037 18589 consensus_queue.cc:1035] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23f6fa8e7f64137b437b99e0477c946" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:54.680413 18587 consensus_queue.cc:1035] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf93444b3394a0cbe4584a810301411" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:54.692444 18592 mvcc.cc:205] Tried to move back new op lower bound from 7298558831314628608 to 7298558831131500544. Current Snapshot: MvccSnapshot[applied={T|T < 7298558831314628608}]
I20260619 13:11:54.697456 18593 mvcc.cc:205] Tried to move back new op lower bound from 7298558831314628608 to 7298558831131500544. Current Snapshot: MvccSnapshot[applied={T|T < 7298558831314628608 or (T in {7298558831314628608})}]
I20260619 13:11:54.705652 18113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:54.705935 18113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:54.709216 18113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdb16b259a9e4eb6b631cb139dab2ee3: Sending DeleteTablet for 3 replicas of tablet 0c319194853545ccba8734fc7d23f1cf
I20260619 13:11:54.710422 18235 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c319194853545ccba8734fc7d23f1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:54 UTC) from {username='slave'} at 127.0.0.1:58972
I20260619 13:11:54.710417 18366 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c319194853545ccba8734fc7d23f1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:54 UTC) from {username='slave'} at 127.0.0.1:57446
I20260619 13:11:54.710527 18496 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c319194853545ccba8734fc7d23f1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:54 UTC) from {username='slave'} at 127.0.0.1:33084
I20260619 13:11:54.713071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18173
I20260619 13:11:54.720805 18604 tablet_replica.cc:330] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: stopping tablet replica
I20260619 13:11:54.721472 18604 raft_consensus.cc:2229] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:11:54.722321 18604 raft_consensus.cc:2258] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:54.724781 18604 ts_tablet_manager.cc:1905] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:54.725522 18603 tablet_replica.cc:330] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: stopping tablet replica
I20260619 13:11:54.726054 18603 raft_consensus.cc:2229] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:54.726702 18603 raft_consensus.cc:2258] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:54.734560 18603 ts_tablet_manager.cc:1905] T 0c319194853545ccba8734fc7d23f1cf P fcf93444b3394a0cbe4584a810301411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:54.734661 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.714759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.0.97.1:33013)
0619 13:11:54.714994 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.715003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.718482 (+ 3479us) server_negotiation.cc:407] Connection header received
0619 13:11:54.718584 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.718592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.718762 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.718895 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:54.724780 (+ 5885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.726467 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.729313 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.730281 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.730411 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.732302 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:54.732317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:54.732342 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:54.733400 (+ 1058us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:54.733476 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:54.733508 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:54.733661 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.733966 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.734133 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:11:54.735225 18602 tablet_replica.cc:330] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: stopping tablet replica
I20260619 13:11:54.735668 18602 raft_consensus.cc:2229] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:54.736220 18295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:54.736259 18602 raft_consensus.cc:2258] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:54.738400 18602 ts_tablet_manager.cc:1905] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:54.740697 18604 ts_tablet_manager.cc:1918] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:54.740979 18604 log.cc:1196] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/0c319194853545ccba8734fc7d23f1cf
I20260619 13:11:54.741721 18604 ts_tablet_manager.cc:1939] T 0c319194853545ccba8734fc7d23f1cf P 88a69e1ca4da47f28656db018e035c7f: Deleting consensus metadata
I20260619 13:11:54.744046 18098 catalog_manager.cc:4841] TS 88a69e1ca4da47f28656db018e035c7f (127.0.97.3:37131): tablet 0c319194853545ccba8734fc7d23f1cf (table test-table [id=cf04357189884a4096b30d2a87fb4dfe]) successfully deleted
I20260619 13:11:54.746893 18602 ts_tablet_manager.cc:1918] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:54.747117 18602 log.cc:1196] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/0c319194853545ccba8734fc7d23f1cf
I20260619 13:11:54.747759 18602 ts_tablet_manager.cc:1939] T 0c319194853545ccba8734fc7d23f1cf P b23f6fa8e7f64137b437b99e0477c946: Deleting consensus metadata
I20260619 13:11:54.749871 18098 catalog_manager.cc:4841] TS b23f6fa8e7f64137b437b99e0477c946 (127.0.97.2:38869): tablet 0c319194853545ccba8734fc7d23f1cf (table test-table [id=cf04357189884a4096b30d2a87fb4dfe]) successfully deleted
I20260619 13:11:54.846347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18173
W20260619 13:11:54.864084 18100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33013 (error 108)
I20260619 13:11:54.864285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18304
I20260619 13:11:54.866155 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.865401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:58988)
0619 13:11:54.865854 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.865989 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":232,"threads_started":1}
W20260619 13:11:54.866422 18100 catalog_manager.cc:4584] TS fcf93444b3394a0cbe4584a810301411 (127.0.97.1:33013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c319194853545ccba8734fc7d23f1cf: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
I20260619 13:11:54.872488 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.865125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.0.97.2:38869)
0619 13:11:54.865313 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:54.865319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:54.865339 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:11:54.865532 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:54.865540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:54.865718 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:54.865820 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:54.866347 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.867561 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.869075 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:54.869763 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:54.869897 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:54.870912 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:54.870922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:54.870941 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:54.871737 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:54.871796 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:54.871829 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:54.871944 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:54.872021 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 13:11:54.872127 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:11:54.873245 18426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:54.920428 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.919745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:58992)
0619 13:11:54.920074 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.920206 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260619 13:11:54.925907 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.921189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:58998)
0619 13:11:54.924880 (+ 3691us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.925007 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3535,"thread_start_us":3414,"threads_started":1}
I20260619 13:11:54.986804 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.986091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:59002)
0619 13:11:54.986442 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.986562 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 13:11:54.988318 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:54.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:59014)
0619 13:11:54.988019 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:54.988114 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 13:11:55.024555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18304
I20260619 13:11:55.051509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18434
I20260619 13:11:55.062018 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:55.052682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.97.3:37131)
0619 13:11:55.052882 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:55.052891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:55.052923 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:11:55.053030 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:55.053038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:55.053187 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:55.053321 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:55.053841 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:55.055304 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:55.057163 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:55.058347 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:55.058500 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:55.059839 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:55.059855 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:55.059878 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:11:55.060902 (+ 1024us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:55.060977 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:11:55.061022 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:55.061158 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:55.061332 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 13:11:55.061457 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:11:55.062881 18556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:55.068842 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:55.068336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:59018)
0619 13:11:55.068566 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:55.068669 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:11:55.069998 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:55.069559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:59024)
0619 13:11:55.069805 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:55.069884 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 13:11:55.207921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18434
I20260619 13:11:55.228180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18083
I20260619 13:11:55.229316 18144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:55.338001 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:55.337422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33013 (local address 127.0.0.1:59038)
0619 13:11:55.337646 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:55.337743 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:11:55.338136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18083
2026-06-19T13:11:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3020 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:11:55 dist-test-slave-q11v krb5kdc[18615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:11:55 dist-test-slave-q11v krb5kdc[18615](info): set up 2 sockets
Jun 19 13:11:55 dist-test-slave-q11v krb5kdc[18615](info): commencing operation
krb5kdc: starting...
W20260619 13:11:56.453644 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:11:56 dist-test-slave-q11v krb5kdc[18615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:11:56.698441 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33787
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:57.034680 18628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:57.035002 18628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:57.035175 18628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:57.043141 18628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:11:57.043256 18628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:57.043351 18628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:57.043443 18628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:11:57.043535 18628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:11:57.053939 18628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:57.055313 18628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:57.056696 18628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:57.068727 18633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:57.068722 18636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:57.068766 18634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:57.069934 18628 server_base.cc:1034] running on GCE node
I20260619 13:11:57.070784 18628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:57.072285 18628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:57.073553 18628 hybrid_clock.cc:648] HybridClock initialized: now 1781874717073478 us; error 57 us; skew 500 ppm
I20260619 13:11:57.074019 18628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:57 dist-test-slave-q11v krb5kdc[18615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:57.077425 18628 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:11:57.079113 18628 webserver.cc:460] Webserver started at http://127.0.97.62:36567/ using document root <none> and password file <none>
I20260619 13:11:57.079957 18628 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:57.080101 18628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:57.080466 18628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:57.083845 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "715276b359434b21841a82b26d678c89"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "fc44ece1cefaa10c705b0130db5d13af"
server_key_iv: "7b75c55aaf46d164286e9b35efb13540"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.084776 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "715276b359434b21841a82b26d678c89"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "fc44ece1cefaa10c705b0130db5d13af"
server_key_iv: "7b75c55aaf46d164286e9b35efb13540"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.090824 18628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:11:57.094547 18643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:57.096155 18628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:11:57.096369 18628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "715276b359434b21841a82b26d678c89"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "fc44ece1cefaa10c705b0130db5d13af"
server_key_iv: "7b75c55aaf46d164286e9b35efb13540"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.096590 18628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:57.122193 18628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:57.126933 18628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:57.127271 18628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:57.148669 18628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33787
I20260619 13:11:57.148686 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33787 every 8 connection(s)
I20260619 13:11:57.150749 18628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:11:57.153347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18628
I20260619 13:11:57.153542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:11:57.154361 388 external_mini_cluster.cc:1428] Setting key d66ec6cbe4d08b265a712b1af1773985
I20260619 13:11:57.155817 18695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:57.172221 18695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: Bootstrap starting.
Jun 19 13:11:57 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:57.176501 18695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:57.177933 18695 log.cc:824] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:57.182061 18695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: No bootstrap required, opened a new log
I20260619 13:11:57.187115 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:57.157381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.0.97.62:33787)
0619 13:11:57.161734 (+ 4353us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:57.161753 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:57.161815 (+ 62us) server_negotiation.cc:407] Connection header received
0619 13:11:57.163934 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:57.163968 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:57.164555 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:57.165088 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:57.167009 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.168335 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.170340 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.171694 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.171918 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:57.177833 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:57.177844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:57.177867 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:57.177937 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:57.182487 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:57.183481 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:57.183489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:57.183499 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:57.183696 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:57.184142 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:57.184145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:57.184147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:57.184744 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:57.184986 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:57.185474 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 13:11:57.186003 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4045,"thread_start_us":3815,"threads_started":1}
I20260619 13:11:57.191615 18695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } }
I20260619 13:11:57.192004 18695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:57.192112 18695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715276b359434b21841a82b26d678c89, State: Initialized, Role: FOLLOWER
I20260619 13:11:57.192660 18695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } }
I20260619 13:11:57.192907 18695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:11:57.193051 18695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:11:57.193224 18695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:57.195752 18695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } }
I20260619 13:11:57.196255 18695 leader_election.cc:302] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 715276b359434b21841a82b26d678c89; no voters:
I20260619 13:11:57.196849 18695 leader_election.cc:288] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:11:57.197057 18700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:57.197837 18700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [term 1 LEADER]: Becoming Leader. State: Replica: 715276b359434b21841a82b26d678c89, State: Running, Role: LEADER
I20260619 13:11:57.198529 18700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } }
I20260619 13:11:57.199280 18695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:11:57.202315 18701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "715276b359434b21841a82b26d678c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } } }
I20260619 13:11:57.202216 18702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 715276b359434b21841a82b26d678c89. Latest consensus state: current_term: 1 leader_uuid: "715276b359434b21841a82b26d678c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715276b359434b21841a82b26d678c89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33787 } } }
I20260619 13:11:57.202728 18702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:57.204384 18701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89 [sys.catalog]: This master's current role is: LEADER
I20260619 13:11:57.204556 18709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:11:57.210178 18709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:11:57.219381 18709 catalog_manager.cc:1347] Generated new cluster ID: 06e1bc162095407bac54dc7f35b19826
I20260619 13:11:57.219518 18709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:11:57.230945 18709 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:11:57.232452 18709 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:11:57.241551 18709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: Generated new TSK 0
I20260619 13:11:57.242578 18709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:11:57.308892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:57.684798 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:57.685105 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:57.685200 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:57.692839 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:57.692957 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:57.693074 18721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:11:57.703701 18721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:57.705348 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:57.706835 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:57.720583 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:57.720669 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:57.720618 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:57.721832 18721 server_base.cc:1034] running on GCE node
I20260619 13:11:57.722445 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:57.723500 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:57.724761 18721 hybrid_clock.cc:648] HybridClock initialized: now 1781874717724712 us; error 43 us; skew 500 ppm
I20260619 13:11:57.725134 18721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:57 dist-test-slave-q11v krb5kdc[18615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:57.728479 18721 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:11:57.730063 18721 webserver.cc:460] Webserver started at http://127.0.97.1:45229/ using document root <none> and password file <none>
I20260619 13:11:57.730770 18721 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:57.730911 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:57.731232 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:57.734247 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "61493f5156ed4e1bbf2204cc95a824e3"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "5854dd2e8518a2616b143d358840aff4"
server_key_iv: "f35ac9df52c0179daf7e6c6a6a19b24c"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.735054 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "61493f5156ed4e1bbf2204cc95a824e3"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "5854dd2e8518a2616b143d358840aff4"
server_key_iv: "f35ac9df52c0179daf7e6c6a6a19b24c"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.740311 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:11:57.744032 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:57.745437 18721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:11:57.745690 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "61493f5156ed4e1bbf2204cc95a824e3"
format_stamp: "Formatted at 2026-06-19 13:11:57 on dist-test-slave-q11v"
server_key: "5854dd2e8518a2616b143d358840aff4"
server_key_iv: "f35ac9df52c0179daf7e6c6a6a19b24c"
server_key_version: "encryptionkey@0"
I20260619 13:11:57.745909 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:57.763934 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:57.773486 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:57.773857 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:57.774866 18721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:57.776253 18721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:57.776362 18721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:57.776521 18721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:57.776600 18721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:57.811687 18721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33685
I20260619 13:11:57.811779 18848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33685 every 8 connection(s)
I20260619 13:11:57.813005 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:11:57.818761 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18721
I20260619 13:11:57.818941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:11:57.819415 388 external_mini_cluster.cc:1428] Setting key 727ef704af32884b413e171fa26a85de
Jun 19 13:11:57 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:57.835500 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:57.815842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33177 (local address 127.0.97.62:33787)
0619 13:11:57.816344 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:57.816352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:57.817333 (+ 981us) server_negotiation.cc:407] Connection header received
0619 13:11:57.819454 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:57.819463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:57.819558 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:57.819686 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:57.821459 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.822723 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.824369 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.825142 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.825296 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:57.829010 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:57.829021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:57.829029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:57.829080 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:57.832048 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:57.833019 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:57.833023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:57.833026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:57.833139 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:57.833704 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:57.833707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:57.833709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:57.834031 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:57.834142 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:57.835052 (+ 910us) server_negotiation.cc:299] Negotiation successful
0619 13:11:57.835227 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260619 13:11:57.837080 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:57.816411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33787 (local address 127.0.97.1:33177)
0619 13:11:57.817155 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:57.817195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:57.819156 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:57.819938 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:57.819948 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:57.820649 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:57.821243 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:57.821263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.822901 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.822906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:57.824207 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:57.824220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:57.825380 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:57.825385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:57.825669 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:57.826478 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:57.826507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:57.828755 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:57.832207 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:57.832220 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:57.832242 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:57.832895 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:57.833247 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:57.833254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:57.833260 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:57.833520 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:57.834166 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:57.834177 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:57.834661 (+ 484us) client_negotiation.cc:769] Sending connection context
0619 13:11:57.834970 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 13:11:57.835560 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":279,"threads_started":1}
I20260619 13:11:57.838850 18849 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33787
I20260619 13:11:57.839196 18849 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:57.839939 18849 heartbeater.cc:511] Master 127.0.97.62:33787 requested a full tablet report, sending...
I20260619 13:11:57.842151 18660 ts_manager.cc:194] Registered new tserver with Master: 61493f5156ed4e1bbf2204cc95a824e3 (127.0.97.1:33685)
I20260619 13:11:57.843634 18660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33177
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:11:57.876680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:58.212492 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:58.212816 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:58.212988 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:58.220144 18855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:58.220309 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:58.220501 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:11:58.230105 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:58.231441 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:58.232796 18855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:58.246570 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:58.246675 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:58.246691 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:58.247454 18855 server_base.cc:1034] running on GCE node
I20260619 13:11:58.248051 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:58.249069 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:58.250300 18855 hybrid_clock.cc:648] HybridClock initialized: now 1781874718250246 us; error 51 us; skew 500 ppm
I20260619 13:11:58.250617 18855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:58 dist-test-slave-q11v krb5kdc[18615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:58.253748 18855 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:11:58.255264 18855 webserver.cc:460] Webserver started at http://127.0.97.2:33681/ using document root <none> and password file <none>
I20260619 13:11:58.256007 18855 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:58.256160 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:58.256474 18855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:58.259573 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "1563346e63cc49a08b7ab69403100e80"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "283a9e656c461236e43dc1a8958579e4"
server_key_iv: "860258111fd4997cda1964d71da0bba4"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.260494 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1563346e63cc49a08b7ab69403100e80"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "283a9e656c461236e43dc1a8958579e4"
server_key_iv: "860258111fd4997cda1964d71da0bba4"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.265933 18855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:11:58.269770 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:58.271267 18855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:11:58.271463 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "1563346e63cc49a08b7ab69403100e80"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "283a9e656c461236e43dc1a8958579e4"
server_key_iv: "860258111fd4997cda1964d71da0bba4"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.271687 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:58.290511 18855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:58.294620 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:58.294914 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:58.295917 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:58.297276 18855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:58.297382 18855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:58.297536 18855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:58.297633 18855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 13:11:58.331688 18855 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40035
I20260619 13:11:58.331732 18982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40035 every 8 connection(s)
I20260619 13:11:58.332983 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:11:58.343040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18855
I20260619 13:11:58.343237 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:11:58.343725 388 external_mini_cluster.cc:1428] Setting key 0210b44f466c381cce17eb82bfaf53ce
Jun 19 13:11:58 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:58.352892 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.335487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45285 (local address 127.0.97.62:33787)
0619 13:11:58.335682 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.335686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.336834 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 13:11:58.338729 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.338734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.338827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.338939 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:58.340454 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.341152 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.342740 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.343211 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.343341 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.346999 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:58.347004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:58.347007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:58.347039 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:58.349849 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.350648 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.350660 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.350663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.350791 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.351215 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.351224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.351226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.351472 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:58.351629 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.352320 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.352547 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:11:58.353981 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.335982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33787 (local address 127.0.97.2:45285)
0619 13:11:58.336666 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:58.336702 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:58.338463 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:58.339193 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:58.339202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:58.339738 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:58.340281 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.340298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.341297 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.341304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.342589 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.342601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.343426 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.343433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.343654 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.344353 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:58.344378 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:58.346772 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:58.349986 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:58.349997 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:58.350012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:58.350522 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:58.350902 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:58.350908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:58.350912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:58.351093 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:58.351598 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:58.351606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:58.351951 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 13:11:58.352223 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 13:11:58.352758 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260619 13:11:58.355422 18983 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33787
I20260619 13:11:58.355741 18983 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:58.356472 18983 heartbeater.cc:511] Master 127.0.97.62:33787 requested a full tablet report, sending...
I20260619 13:11:58.358021 18660 ts_manager.cc:194] Registered new tserver with Master: 1563346e63cc49a08b7ab69403100e80 (127.0.97.2:40035)
I20260619 13:11:58.359289 18660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45285
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:11:58.397178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:11:58.723483 18988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:11:58.723781 18988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:11:58.723875 18988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:11:58.731009 18988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:11:58.731142 18988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:11:58.731256 18988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:11:58.741510 18988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:11:58.742785 18988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:11:58.744027 18988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:11:58.757091 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:58.757064 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:11:58.757339 18996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:11:58.757941 18988 server_base.cc:1034] running on GCE node
I20260619 13:11:58.758662 18988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:11:58.759702 18988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:11:58.760944 18988 hybrid_clock.cc:648] HybridClock initialized: now 1781874718760883 us; error 49 us; skew 500 ppm
I20260619 13:11:58.761334 18988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:11:58 dist-test-slave-q11v krb5kdc[18615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:11:58.764652 18988 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:11:58.766184 18988 webserver.cc:460] Webserver started at http://127.0.97.3:40587/ using document root <none> and password file <none>
I20260619 13:11:58.766883 18988 fs_manager.cc:359] Metadata directory not provided
I20260619 13:11:58.767030 18988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:11:58.767346 18988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:11:58.771075 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "32949c8ca7f74626a9120dafe3d8d5f9"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "a27413341f292b7910465d19a3b5c8ab"
server_key_iv: "a8e822b3a24ea93c2a4ca84935b0ddfa"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.771943 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "32949c8ca7f74626a9120dafe3d8d5f9"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "a27413341f292b7910465d19a3b5c8ab"
server_key_iv: "a8e822b3a24ea93c2a4ca84935b0ddfa"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.777683 18988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:11:58.781354 19003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:58.782792 18988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:11:58.782980 18988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "32949c8ca7f74626a9120dafe3d8d5f9"
format_stamp: "Formatted at 2026-06-19 13:11:58 on dist-test-slave-q11v"
server_key: "a27413341f292b7910465d19a3b5c8ab"
server_key_iv: "a8e822b3a24ea93c2a4ca84935b0ddfa"
server_key_version: "encryptionkey@0"
I20260619 13:11:58.783201 18988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:11:58.795866 18988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:11:58.800021 18988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:11:58.800376 18988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:11:58.801396 18988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:11:58.802798 18988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:11:58.802902 18988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:58.803054 18988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:11:58.803126 18988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:11:58.837540 18988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42639
I20260619 13:11:58.837579 19115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42639 every 8 connection(s)
I20260619 13:11:58.838836 18988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:11:58.841424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 18988
I20260619 13:11:58.841647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:11:58.842087 388 external_mini_cluster.cc:1428] Setting key 885e391e350301533a6c7733899fe281
I20260619 13:11:58.846764 18849 heartbeater.cc:503] Master 127.0.97.62:33787 was elected leader, sending a full tablet report...
Jun 19 13:11:58 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:11:58.859491 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.841499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58869 (local address 127.0.97.62:33787)
0619 13:11:58.841729 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.841734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.843050 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 13:11:58.845320 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.845324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.845387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.845468 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:58.847042 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.848035 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.849794 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.850236 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.850332 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.853572 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:58.853635 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:58.853641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:58.853681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:58.856732 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.857442 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.857445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.857448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.857645 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.858034 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.858037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.858039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.858309 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:58.858416 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.859055 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.859235 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 13:11:58.860500 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.842030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33787 (local address 127.0.97.3:58869)
0619 13:11:58.842852 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:58.842917 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:58.845080 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:58.845737 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:58.845744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:58.846271 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:11:58.846856 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.846872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.848205 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.848211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.849659 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.849677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.850418 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.850423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.850638 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.851225 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:11:58.851246 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:11:58.853364 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:11:58.856874 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:58.856883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:58.856899 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:58.857327 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:58.857752 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:11:58.857755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:11:58.857757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:11:58.857931 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:11:58.858408 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:11:58.858413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:11:58.858739 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 13:11:58.858952 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 13:11:58.859345 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":296,"threads_started":1}
I20260619 13:11:58.862015 19116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33787
I20260619 13:11:58.862296 19116 heartbeater.cc:464] Registering TS with master...
I20260619 13:11:58.862954 19116 heartbeater.cc:511] Master 127.0.97.62:33787 requested a full tablet report, sending...
I20260619 13:11:58.864351 18660 ts_manager.cc:194] Registered new tserver with Master: 32949c8ca7f74626a9120dafe3d8d5f9 (127.0.97.3:42639)
I20260619 13:11:58.865226 18660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58869
I20260619 13:11:58.869282 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:11:58.886492 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.873363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.0.97.62:33787)
0619 13:11:58.873518 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.873523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.874008 (+ 485us) server_negotiation.cc:407] Connection header received
0619 13:11:58.874144 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.874148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.874208 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.874287 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:58.875827 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.876560 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.877871 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.878389 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.878497 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.881075 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:58.881080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:58.881083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:58.881116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:58.883843 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.884563 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.884567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.884569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.884691 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:58.885048 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:58.885051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:58.885053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:58.885333 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:58.885549 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.885941 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.886083 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:11:58.890602 18660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:11:58.897552 18660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:11:58.924279 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.910277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40035 (local address 127.0.0.1:50734)
0619 13:11:58.915965 (+ 5688us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:58.915984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:58.916240 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:58.917201 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:58.917210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:58.917244 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:58.918184 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.918196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.919687 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.919691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.921270 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.921281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.923240 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.923246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.923465 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.923526 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 13:11:58.923666 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 13:11:58.923863 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5560,"mutex_wait_us":3,"thread_start_us":5148,"threads_started":1}
I20260619 13:11:58.924295 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.911453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.97.2:40035)
0619 13:11:58.912165 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.912180 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.916102 (+ 3922us) server_negotiation.cc:407] Connection header received
0619 13:11:58.916492 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.916500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.916724 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.916856 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:58.918400 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.919531 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.921429 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.923032 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.923166 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.923416 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.923766 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.923955 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":339,"threads_started":1}
I20260619 13:11:58.925760 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.910221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33685 (local address 127.0.0.1:54590)
0619 13:11:58.915836 (+ 5615us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:58.915880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:58.916204 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:58.917286 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:58.917290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:58.917320 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:58.917832 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.917857 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.920486 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.920494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.922881 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.922895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.924890 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.924896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.925079 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.925111 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:11:58.925189 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:11:58.925328 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5346,"thread_start_us":5462,"threads_started":1}
I20260619 13:11:58.925808 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.910994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.97.1:33685)
0619 13:11:58.915477 (+ 4483us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.915490 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.916015 (+ 525us) server_negotiation.cc:407] Connection header received
0619 13:11:58.916469 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.916479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.916745 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.916910 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:58.918107 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.920254 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.923111 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.924685 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.924820 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.925058 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.925275 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.925442 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4237,"thread_start_us":4021,"threads_started":1}
I20260619 13:11:58.928908 18918 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (DEFAULT_TABLE table=test-table [id=c454cfc0622e4d94813697237dfdb694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:58.929296 18784 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (DEFAULT_TABLE table=test-table [id=c454cfc0622e4d94813697237dfdb694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:58.930495 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:58.930639 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.917509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42639 (local address 127.0.0.1:37550)
0619 13:11:58.921890 (+ 4381us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:58.921909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:58.922034 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:58.923075 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:58.923083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:58.923116 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:58.923530 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.923540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.925488 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.925493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.928070 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:58.928085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.929855 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.929862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:58.930046 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.930081 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:11:58.930178 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:11:58.930329 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4229,"thread_start_us":4085,"threads_started":1}
I20260619 13:11:58.930751 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.916705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.97.3:42639)
0619 13:11:58.917484 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:58.917493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:58.922179 (+ 4686us) server_negotiation.cc:407] Connection header received
0619 13:11:58.922295 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:58.922306 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:58.922584 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:58.922797 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:58.923707 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.925315 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.928262 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:58.929645 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:58.929781 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:58.929994 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:58.930237 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 13:11:58.930391 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":445,"threads_started":1}
I20260619 13:11:58.930716 18784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:58.934645 19051 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (DEFAULT_TABLE table=test-table [id=c454cfc0622e4d94813697237dfdb694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:11:58.936138 19051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:11:58.942358 19134 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Bootstrap starting.
I20260619 13:11:58.944921 19135 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Bootstrap starting.
I20260619 13:11:58.945848 19134 tablet_bootstrap.cc:654] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:58.947144 19134 log.cc:824] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:58.948278 19135 tablet_bootstrap.cc:654] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:58.949352 19137 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Bootstrap starting.
I20260619 13:11:58.949559 19135 log.cc:824] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:58.951701 19134 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: No bootstrap required, opened a new log
I20260619 13:11:58.952035 19134 ts_tablet_manager.cc:1397] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:11:58.952704 19137 tablet_bootstrap.cc:654] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Neither blocks nor log segments found. Creating new log.
I20260619 13:11:58.953649 19137 log.cc:824] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:11:58.956075 19135 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: No bootstrap required, opened a new log
I20260619 13:11:58.956357 19135 ts_tablet_manager.cc:1397] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 13:11:58.956626 19137 tablet_bootstrap.cc:492] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: No bootstrap required, opened a new log
I20260619 13:11:58.956872 19137 ts_tablet_manager.cc:1397] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 13:11:58.960280 19134 raft_consensus.cc:348] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.960757 19134 raft_consensus.cc:374] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:58.960883 19134 raft_consensus.cc:729] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61493f5156ed4e1bbf2204cc95a824e3, State: Initialized, Role: FOLLOWER
I20260619 13:11:58.961938 19134 consensus_queue.cc:260] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.963728 19134 ts_tablet_manager.cc:1428] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 13:11:58.967017 19135 raft_consensus.cc:348] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.967463 19135 raft_consensus.cc:374] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:58.967255 19137 raft_consensus.cc:348] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.967577 19135 raft_consensus.cc:729] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1563346e63cc49a08b7ab69403100e80, State: Initialized, Role: FOLLOWER
I20260619 13:11:58.967700 19137 raft_consensus.cc:374] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:11:58.967854 19137 raft_consensus.cc:729] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32949c8ca7f74626a9120dafe3d8d5f9, State: Initialized, Role: FOLLOWER
I20260619 13:11:58.968307 19135 consensus_queue.cc:260] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.968591 19137 consensus_queue.cc:260] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.969791 18983 heartbeater.cc:503] Master 127.0.97.62:33787 was elected leader, sending a full tablet report...
I20260619 13:11:58.970263 19116 heartbeater.cc:503] Master 127.0.97.62:33787 was elected leader, sending a full tablet report...
I20260619 13:11:58.970327 19135 ts_tablet_manager.cc:1428] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 13:11:58.970902 19137 ts_tablet_manager.cc:1428] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260619 13:11:58.997184 19142 raft_consensus.cc:482] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:11:58.997512 19142 raft_consensus.cc:504] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:58.999502 19142 leader_election.cc:288] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61493f5156ed4e1bbf2204cc95a824e3 (127.0.97.1:33685), 1563346e63cc49a08b7ab69403100e80 (127.0.97.2:40035)
I20260619 13:11:59.005662 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.999908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54993 (local address 127.0.97.1:33685)
0619 13:11:59.000120 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.000127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.000629 (+ 502us) server_negotiation.cc:407] Connection header received
0619 13:11:59.000710 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.000714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.000796 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.000890 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:59.001376 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.002161 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.003834 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.004727 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.004876 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.004978 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.005243 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.005371 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 13:11:59.005681 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.999759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33685 (local address 127.0.97.3:54993)
0619 13:11:59.000290 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:59.000311 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:59.000481 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:59.000932 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:59.000936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:59.001008 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:59.001191 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:59.001220 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.002305 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.002310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:59.003673 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:59.003684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.004885 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.004891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:59.005084 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.005108 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:11:59.005216 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:11:59.005358 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":23424}
I20260619 13:11:59.006063 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:59.000104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51479 (local address 127.0.97.2:40035)
0619 13:11:59.000323 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.000331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.000940 (+ 609us) server_negotiation.cc:407] Connection header received
0619 13:11:59.001103 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.001107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.001163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.001286 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:11:59.001831 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.002526 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.004335 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.005118 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.005274 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.005371 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.005688 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.005817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:11:59.006150 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:58.999901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40035 (local address 127.0.97.3:51479)
0619 13:11:59.000814 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 13:11:59.000832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:11:59.000974 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:11:59.001304 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:11:59.001308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:11:59.001331 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:11:59.001679 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:59.001690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.002678 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.002684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:59.004183 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:11:59.004216 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.005326 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.005332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:11:59.005503 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.005529 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 13:11:59.005663 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 13:11:59.005789 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":545,"threads_started":1}
I20260619 13:11:59.007114 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa64d92c4634fbbaa95a53ab8c2dfa7" candidate_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1563346e63cc49a08b7ab69403100e80" is_pre_election: true
I20260619 13:11:59.007105 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa64d92c4634fbbaa95a53ab8c2dfa7" candidate_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61493f5156ed4e1bbf2204cc95a824e3" is_pre_election: true
I20260619 13:11:59.007731 18804 raft_consensus.cc:2454] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32949c8ca7f74626a9120dafe3d8d5f9 in term 0.
I20260619 13:11:59.007731 18938 raft_consensus.cc:2454] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32949c8ca7f74626a9120dafe3d8d5f9 in term 0.
I20260619 13:11:59.008543 19007 leader_election.cc:302] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1563346e63cc49a08b7ab69403100e80, 32949c8ca7f74626a9120dafe3d8d5f9, 61493f5156ed4e1bbf2204cc95a824e3; no voters:
I20260619 13:11:59.008989 19142 raft_consensus.cc:2781] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:11:59.009145 19142 raft_consensus.cc:482] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:11:59.009250 19142 raft_consensus.cc:3037] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:59.011906 19142 raft_consensus.cc:504] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:59.012660 19142 leader_election.cc:288] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [CANDIDATE]: Term 1 election: Requested vote from peers 61493f5156ed4e1bbf2204cc95a824e3 (127.0.97.1:33685), 1563346e63cc49a08b7ab69403100e80 (127.0.97.2:40035)
I20260619 13:11:59.013128 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa64d92c4634fbbaa95a53ab8c2dfa7" candidate_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61493f5156ed4e1bbf2204cc95a824e3"
I20260619 13:11:59.013221 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa64d92c4634fbbaa95a53ab8c2dfa7" candidate_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1563346e63cc49a08b7ab69403100e80"
I20260619 13:11:59.013388 18804 raft_consensus.cc:3037] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:59.013487 18938 raft_consensus.cc:3037] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:11:59.015751 18938 raft_consensus.cc:2454] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32949c8ca7f74626a9120dafe3d8d5f9 in term 1.
I20260619 13:11:59.015770 18804 raft_consensus.cc:2454] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32949c8ca7f74626a9120dafe3d8d5f9 in term 1.
I20260619 13:11:59.016253 19004 leader_election.cc:302] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1563346e63cc49a08b7ab69403100e80, 32949c8ca7f74626a9120dafe3d8d5f9; no voters:
I20260619 13:11:59.016599 19142 raft_consensus.cc:2781] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:11:59.017122 19142 raft_consensus.cc:686] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 32949c8ca7f74626a9120dafe3d8d5f9, State: Running, Role: LEADER
I20260619 13:11:59.017670 19142 consensus_queue.cc:237] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } }
I20260619 13:11:59.022394 18660 catalog_manager.cc:5495] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32949c8ca7f74626a9120dafe3d8d5f9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32949c8ca7f74626a9120dafe3d8d5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260619 13:11:59.051765 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:59.044563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.97.3:42639)
0619 13:11:59.045007 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.045014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.045038 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:11:59.045212 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.045220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.045380 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.045566 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:11:59.046002 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.046729 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.048291 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.048849 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.048978 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.050211 (+ 1233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:11:59.051031 (+ 820us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:11:59.051238 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.051392 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.051450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260619 13:11:59.063066 18804 raft_consensus.cc:1261] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 1 FOLLOWER]: Refusing update from remote peer 32949c8ca7f74626a9120dafe3d8d5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:59.063191 18938 raft_consensus.cc:1261] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 1 FOLLOWER]: Refusing update from remote peer 32949c8ca7f74626a9120dafe3d8d5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:11:59.064232 19144 consensus_queue.cc:1035] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61493f5156ed4e1bbf2204cc95a824e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:11:59.064673 19142 consensus_queue.cc:1035] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1563346e63cc49a08b7ab69403100e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:11:59.065915 18850 tablet.cc:2369] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:11:59.085049 18984 tablet.cc:2369] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:11:59.085894 19149 mvcc.cc:205] Tried to move back new op lower bound from 7298558849270185984 to 7298558849100673024. Current Snapshot: MvccSnapshot[applied={T|T < 7298558849270185984}]
I20260619 13:11:59.100448 18658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:59.100814 18658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:11:59.104772 18658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 715276b359434b21841a82b26d678c89: Sending DeleteTablet for 3 replicas of tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7
I20260619 13:11:59.106074 18784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54590
I20260619 13:11:59.106518 18918 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50734
I20260619 13:11:59.107240 19051 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:11:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37550
I20260619 13:11:59.110440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18721
I20260619 13:11:59.117724 19158 tablet_replica.cc:330] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: stopping tablet replica
I20260619 13:11:59.118327 19158 raft_consensus.cc:2229] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:59.119011 19158 raft_consensus.cc:2258] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:59.119911 19159 tablet_replica.cc:330] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: stopping tablet replica
I20260619 13:11:59.120169 19157 tablet_replica.cc:330] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: stopping tablet replica
I20260619 13:11:59.120340 19159 raft_consensus.cc:2229] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:11:59.120551 19157 raft_consensus.cc:2229] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:11:59.120998 19159 raft_consensus.cc:2258] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:59.121073 19157 raft_consensus.cc:2258] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:11:59.123478 19158 ts_tablet_manager.cc:1905] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:59.124992 19157 ts_tablet_manager.cc:1905] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:59.126914 19159 ts_tablet_manager.cc:1905] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:11:59.132967 19158 ts_tablet_manager.cc:1918] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:59.133244 19158 log.cc:1196] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/0fa64d92c4634fbbaa95a53ab8c2dfa7
I20260619 13:11:59.133955 19158 ts_tablet_manager.cc:1939] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 1563346e63cc49a08b7ab69403100e80: Deleting consensus metadata
Jun 19 13:11:59 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:11:59.136168 18644 catalog_manager.cc:4841] TS 1563346e63cc49a08b7ab69403100e80 (127.0.97.2:40035): tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (table test-table [id=c454cfc0622e4d94813697237dfdb694]) successfully deleted
I20260619 13:11:59.136847 19157 ts_tablet_manager.cc:1918] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:59.137046 19157 log.cc:1196] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/0fa64d92c4634fbbaa95a53ab8c2dfa7
I20260619 13:11:59.137657 19157 ts_tablet_manager.cc:1939] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 61493f5156ed4e1bbf2204cc95a824e3: Deleting consensus metadata
I20260619 13:11:59.139699 18647 catalog_manager.cc:4841] TS 61493f5156ed4e1bbf2204cc95a824e3 (127.0.97.1:33685): tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (table test-table [id=c454cfc0622e4d94813697237dfdb694]) successfully deleted
I20260619 13:11:59.142194 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:59.112117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.0.97.1:33685)
0619 13:11:59.112357 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.112367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.123903 (+ 11536us) server_negotiation.cc:407] Connection header received
0619 13:11:59.124133 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.124139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.124234 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.124336 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:59.125952 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.127051 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.128976 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.129583 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.129715 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.135399 (+ 5684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:59.135413 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:59.135432 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:59.135480 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:59.139008 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.140001 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.140009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.140011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.140141 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.140549 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.140553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.140556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.140881 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:59.141357 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.141559 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.141806 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:11:59.142820 18844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:59.144542 19159 ts_tablet_manager.cc:1918] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:11:59.144723 19159 log.cc:1196] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/0fa64d92c4634fbbaa95a53ab8c2dfa7
I20260619 13:11:59.145295 19159 ts_tablet_manager.cc:1939] T 0fa64d92c4634fbbaa95a53ab8c2dfa7 P 32949c8ca7f74626a9120dafe3d8d5f9: Deleting consensus metadata
I20260619 13:11:59.147109 18644 catalog_manager.cc:4841] TS 32949c8ca7f74626a9120dafe3d8d5f9 (127.0.97.3:42639): tablet 0fa64d92c4634fbbaa95a53ab8c2dfa7 (table test-table [id=c454cfc0622e4d94813697237dfdb694]) successfully deleted
I20260619 13:11:59.245134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18721
I20260619 13:11:59.263814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18855
Jun 19 13:11:59 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:11:59.281678 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:59.264668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.97.2:40035)
0619 13:11:59.264863 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.264869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.264891 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:11:59.265018 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.265024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.265124 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.265243 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:59.266501 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.267646 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.268990 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.269679 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.269813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.274251 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:59.274275 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:59.274295 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:59.274344 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:59.278458 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.279300 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.279311 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.279314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.279481 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.279846 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.279851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.279854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.280345 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:59.280673 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.281064 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.281295 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:11:59.282435 18978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:59.372627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18855
I20260619 13:11:59.390938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18988
Jun 19 13:11:59 dist-test-slave-q11v krb5kdc[18615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:11:59.410118 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:11:59.391777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.97.3:42639)
0619 13:11:59.391985 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:11:59.391991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:11:59.392013 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:11:59.392133 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:11:59.392139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:11:59.392247 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:11:59.392378 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:11:59.393753 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.395072 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.396772 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:11:59.397502 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:11:59.397732 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:11:59.403061 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:11:59.403082 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:11:59.403106 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:11:59.403161 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:11:59.406684 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.407603 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.407611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.407613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.407778 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:11:59.408245 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:11:59.408250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:11:59.408252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:11:59.408691 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:11:59.408951 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 13:11:59.409386 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 13:11:59.409585 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:11:59.410872 19111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:59.533380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18988
I20260619 13:11:59.552922 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 18628
I20260619 13:11:59.554149 18690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:11:59.664944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 18628
2026-06-19T13:11:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4327 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T13:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:11:59Z Disabled control of system clock
I20260619 13:11:59.742771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34983
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:12:00.078226 19168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:00.078548 19168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:00.078689 19168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:00.086218 19168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:00.086334 19168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:00.086411 19168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:00.086531 19168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:00.086611 19168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:00.096246 19168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:00.097681 19168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:00.099566 19168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:00.111620 19173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:00.111640 19176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:00.111714 19174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:00.113155 19168 server_base.cc:1034] running on GCE node
I20260619 13:12:00.114099 19168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:00.115655 19168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:00.116928 19168 hybrid_clock.cc:648] HybridClock initialized: now 1781874720116844 us; error 75 us; skew 500 ppm
I20260619 13:12:00.117384 19168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:00.120201 19168 webserver.cc:460] Webserver started at http://127.0.97.62:41247/ using document root <none> and password file <none>
I20260619 13:12:00.120983 19168 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:00.121135 19168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:00.121498 19168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:00.124866 19168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "266dcaaff0a743deb263de7d3b4b3251"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "706192e77a695c367feb0b7eb4a8221c"
server_key_iv: "d7f1f21c03f227a39a2130ffb9072973"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.125856 19168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "266dcaaff0a743deb263de7d3b4b3251"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "706192e77a695c367feb0b7eb4a8221c"
server_key_iv: "d7f1f21c03f227a39a2130ffb9072973"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.131659 19168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:12:00.135525 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:00.137068 19168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:12:00.137279 19168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "266dcaaff0a743deb263de7d3b4b3251"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "706192e77a695c367feb0b7eb4a8221c"
server_key_iv: "d7f1f21c03f227a39a2130ffb9072973"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.137531 19168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:00.174324 19168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:00.175398 19168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:00.175725 19168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:00.196223 19168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34983
I20260619 13:12:00.196229 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34983 every 8 connection(s)
I20260619 13:12:00.197827 19168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:00.201713 19234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:00.206218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19168
I20260619 13:12:00.206431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:00.206954 388 external_mini_cluster.cc:1428] Setting key 5a4bb8cd5043761c55c121549e820836
I20260619 13:12:00.214334 19234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: Bootstrap starting.
I20260619 13:12:00.218181 19234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:00.219565 19234 log.cc:824] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:00.222749 19234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: No bootstrap required, opened a new log
I20260619 13:12:00.223598 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:00.209057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.97.62:34983)
0619 13:12:00.210258 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:00.210279 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:00.210330 (+ 51us) server_negotiation.cc:407] Connection header received
0619 13:12:00.212865 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:00.212895 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:00.213513 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:00.214006 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:00.214573 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.216569 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.218142 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.219235 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.219443 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:00.220908 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:00.220917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:00.220933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:00.221506 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:00.221644 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:00.221684 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:00.221828 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:00.222081 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 13:12:00.222412 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":358,"threads_started":1}
I20260619 13:12:00.231066 19234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } }
I20260619 13:12:00.231531 19234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:00.231652 19234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266dcaaff0a743deb263de7d3b4b3251, State: Initialized, Role: FOLLOWER
I20260619 13:12:00.232237 19234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } }
I20260619 13:12:00.232510 19234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:00.232654 19234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:00.232883 19234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:00.235716 19234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } }
I20260619 13:12:00.236235 19234 leader_election.cc:302] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266dcaaff0a743deb263de7d3b4b3251; no voters:
I20260619 13:12:00.236887 19234 leader_election.cc:288] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:00.237006 19239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:00.238729 19239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [term 1 LEADER]: Becoming Leader. State: Replica: 266dcaaff0a743deb263de7d3b4b3251, State: Running, Role: LEADER
I20260619 13:12:00.239451 19234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:00.239287 19239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } }
I20260619 13:12:00.242457 19240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266dcaaff0a743deb263de7d3b4b3251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } } }
I20260619 13:12:00.242501 19241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266dcaaff0a743deb263de7d3b4b3251. Latest consensus state: current_term: 1 leader_uuid: "266dcaaff0a743deb263de7d3b4b3251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266dcaaff0a743deb263de7d3b4b3251" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34983 } } }
I20260619 13:12:00.242901 19240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:00.243017 19241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:00.244123 19247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:00.250834 19247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:00.259428 19247 catalog_manager.cc:1347] Generated new cluster ID: 4aed1b010abe45d2a10b96e8c96c3aea
I20260619 13:12:00.259554 19247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:00.278335 19247 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:00.279697 19247 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:00.286717 19247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: Generated new TSK 0
I20260619 13:12:00.287621 19247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:00.299790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--builtin_ntp_servers=127.0.97.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:12:00.657508 19258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:00.657869 19258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:00.657961 19258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:00.665201 19258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:00.665309 19258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:00.665493 19258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:12:00.674897 19258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:00.676187 19258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:00.677953 19258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:00.691198 19263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:00.691221 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:00.691830 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:00.692150 19258 server_base.cc:1034] running on GCE node
I20260619 13:12:00.692783 19258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:00.693862 19258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:00.695099 19258 hybrid_clock.cc:648] HybridClock initialized: now 1781874720695051 us; error 44 us; skew 500 ppm
I20260619 13:12:00.695430 19258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:00.697804 19258 webserver.cc:460] Webserver started at http://127.0.97.1:46411/ using document root <none> and password file <none>
I20260619 13:12:00.698510 19258 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:00.698649 19258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:00.698974 19258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:00.701941 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "d12302e22ea543e0acc03305dc444a3a"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "8beeb7d0aa22f1fca152b0c2df454fe2"
server_key_iv: "0c907f317f2edd8553237c2302ef815d"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.702740 19258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d12302e22ea543e0acc03305dc444a3a"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "8beeb7d0aa22f1fca152b0c2df454fe2"
server_key_iv: "0c907f317f2edd8553237c2302ef815d"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.708108 19258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 13:12:00.711737 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:00.713182 19258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:12:00.713371 19258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "d12302e22ea543e0acc03305dc444a3a"
format_stamp: "Formatted at 2026-06-19 13:12:00 on dist-test-slave-q11v"
server_key: "8beeb7d0aa22f1fca152b0c2df454fe2"
server_key_iv: "0c907f317f2edd8553237c2302ef815d"
server_key_version: "encryptionkey@0"
I20260619 13:12:00.713569 19258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:00.734707 19258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:00.735527 19258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:00.735769 19258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:00.736696 19258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:00.738269 19258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:00.738359 19258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:00.738508 19258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:00.738617 19258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:00.772774 19258 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44895
I20260619 13:12:00.772884 19384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44895 every 8 connection(s)
I20260619 13:12:00.774185 19258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:12:00.779837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19258
I20260619 13:12:00.780050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:12:00.780484 388 external_mini_cluster.cc:1428] Setting key a1c49dfa8008dbd68b789ae8f56f65c8
I20260619 13:12:00.783537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--builtin_ntp_servers=127.0.97.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:12:00.788226 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:00.776800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46285 (local address 127.0.97.62:34983)
0619 13:12:00.777272 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:00.777277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:00.778602 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 13:12:00.780503 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:00.780511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:00.780623 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:00.780743 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:00.782180 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.783215 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.784889 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.785707 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.785851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:00.787273 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:00.787281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:00.787289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:00.787314 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:00.787364 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:00.787382 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:00.787488 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:00.787681 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 13:12:00.787796 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260619 13:12:00.789059 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:00.777441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34983 (local address 127.0.97.1:46285)
0619 13:12:00.778433 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:00.778475 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:00.780240 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:00.781010 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:00.781031 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:00.781158 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:00.781968 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:00.781991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.783366 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.783370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:00.784740 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:00.784752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:00.785913 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:00.785918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:00.786148 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:00.786920 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:00.786924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:00.786975 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:00.787070 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:00.787481 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:00.787489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:00.787495 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:12:00.787639 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 13:12:00.787945 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260619 13:12:00.790647 19385 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34983
I20260619 13:12:00.790973 19385 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:00.791749 19385 heartbeater.cc:511] Master 127.0.97.62:34983 requested a full tablet report, sending...
I20260619 13:12:00.793860 19199 ts_manager.cc:194] Registered new tserver with Master: d12302e22ea543e0acc03305dc444a3a (127.0.97.1:44895)
I20260619 13:12:00.795331 19199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46285
W20260619 13:12:01.118121 19389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:01.118465 19389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:01.118620 19389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:01.126091 19389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:01.126256 19389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:01.126449 19389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:12:01.136193 19389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:01.137609 19389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:01.139708 19389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:01.153934 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:01.154045 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:01.154045 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:01.154948 19389 server_base.cc:1034] running on GCE node
I20260619 13:12:01.155581 19389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:01.156535 19389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:01.157768 19389 hybrid_clock.cc:648] HybridClock initialized: now 1781874721157731 us; error 56 us; skew 500 ppm
I20260619 13:12:01.158123 19389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:01.160641 19389 webserver.cc:460] Webserver started at http://127.0.97.2:45739/ using document root <none> and password file <none>
I20260619 13:12:01.161363 19389 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:01.161509 19389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:01.161885 19389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:01.164816 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "d96063f9a49729a7b8eda936424a3377"
server_key_iv: "121875d3c695ac5da59c80c7814a9dd8"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.165709 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "d96063f9a49729a7b8eda936424a3377"
server_key_iv: "121875d3c695ac5da59c80c7814a9dd8"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.171130 19389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:12:01.174726 19403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.176151 19389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:12:01.176345 19389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "d96063f9a49729a7b8eda936424a3377"
server_key_iv: "121875d3c695ac5da59c80c7814a9dd8"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.176616 19389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:01.188037 19389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:01.188887 19389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:01.189180 19389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:01.190232 19389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:01.191753 19389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:01.191875 19389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.192015 19389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:01.192116 19389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.228441 19389 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46185
I20260619 13:12:01.228506 19515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46185 every 8 connection(s)
I20260619 13:12:01.229936 19389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:12:01.230423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19389
I20260619 13:12:01.230613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:12:01.231249 388 external_mini_cluster.cc:1428] Setting key f34a49d38ebd038d92c7831c6860195d
I20260619 13:12:01.234575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--builtin_ntp_servers=127.0.97.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:12:01.246012 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.232651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41013 (local address 127.0.97.62:34983)
0619 13:12:01.232868 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.232875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.234610 (+ 1735us) server_negotiation.cc:407] Connection header received
0619 13:12:01.237577 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.237582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.237712 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.237803 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:01.239519 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.240351 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.243024 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.243440 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.243588 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.245046 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:01.245052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:01.245064 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:01.245086 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:01.245124 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:01.245137 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:01.245235 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.245562 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.245691 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:12:01.247382 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.233288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34983 (local address 127.0.97.2:41013)
0619 13:12:01.234441 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.234490 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.237237 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.238096 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.238110 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.238243 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:01.239081 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.239111 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.240535 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.240543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.242848 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.242866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.243697 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.243711 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.243930 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.244709 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:01.244712 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:01.244753 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:01.244836 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:01.245234 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:01.245249 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:01.245259 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.245552 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.245990 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260619 13:12:01.248952 19516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34983
I20260619 13:12:01.249295 19516 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:01.250248 19516 heartbeater.cc:511] Master 127.0.97.62:34983 requested a full tablet report, sending...
I20260619 13:12:01.251906 19199 ts_manager.cc:194] Registered new tserver with Master: cfe50edbd4eb4f7ab63a4e5de7cd45e7 (127.0.97.2:46185)
I20260619 13:12:01.252739 19199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41013
W20260619 13:12:01.582309 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:01.582626 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:01.582767 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:01.590826 19519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:01.590945 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:01.591122 19519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:12:01.600535 19519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:01.601962 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:01.603885 19519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:01.618005 19525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:01.618163 19524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:01.618134 19527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:01.619349 19519 server_base.cc:1034] running on GCE node
I20260619 13:12:01.620090 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:01.621148 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:01.622442 19519 hybrid_clock.cc:648] HybridClock initialized: now 1781874721622350 us; error 54 us; skew 500 ppm
I20260619 13:12:01.622838 19519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:01.625517 19519 webserver.cc:460] Webserver started at http://127.0.97.3:37691/ using document root <none> and password file <none>
I20260619 13:12:01.626559 19519 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:01.626727 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:01.627102 19519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:01.630470 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "48e91b8363dc4c99b2ef2b7c0f444a41"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "8198c63a4cd84abb0cf304b0bdf3904a"
server_key_iv: "1550625dbdc3d6af4a9fba3f53712009"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.631348 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48e91b8363dc4c99b2ef2b7c0f444a41"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "8198c63a4cd84abb0cf304b0bdf3904a"
server_key_iv: "1550625dbdc3d6af4a9fba3f53712009"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.636953 19519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 13:12:01.641031 19533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.642704 19519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:01.642884 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "48e91b8363dc4c99b2ef2b7c0f444a41"
format_stamp: "Formatted at 2026-06-19 13:12:01 on dist-test-slave-q11v"
server_key: "8198c63a4cd84abb0cf304b0bdf3904a"
server_key_iv: "1550625dbdc3d6af4a9fba3f53712009"
server_key_version: "encryptionkey@0"
I20260619 13:12:01.643126 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:01.671736 19519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:01.672680 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:01.672982 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:01.674110 19519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:01.675755 19519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:01.675889 19519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.676023 19519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:01.676113 19519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:01.710717 19519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38719
I20260619 13:12:01.710778 19645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38719 every 8 connection(s)
I20260619 13:12:01.712100 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:12:01.715749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19519
I20260619 13:12:01.715958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:12:01.716617 388 external_mini_cluster.cc:1428] Setting key abb2ec1066f2609126d92e9a97d9ba60
I20260619 13:12:01.726351 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.714906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34635 (local address 127.0.97.62:34983)
0619 13:12:01.715117 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.715121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.716995 (+ 1874us) server_negotiation.cc:407] Connection header received
0619 13:12:01.719644 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.719650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.719745 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.719855 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:01.721094 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.721939 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.723651 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.724072 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.724169 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.725583 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:01.725639 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:01.725645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:01.725675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:01.725706 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:01.725720 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:01.725790 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.726017 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.726100 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:12:01.727451 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.715467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34983 (local address 127.0.97.3:34635)
0619 13:12:01.716788 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.716846 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.719169 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.720166 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.720175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.720276 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:01.720896 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.720913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.722101 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.722105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.723504 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.723517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.724274 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.724279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.724518 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.725239 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:01.725242 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:01.725288 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:01.725372 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:01.725802 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:01.725810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:01.725816 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.725989 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.726299 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":408,"threads_started":1}
I20260619 13:12:01.729008 19646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34983
I20260619 13:12:01.729336 19646 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:01.730253 19646 heartbeater.cc:511] Master 127.0.97.62:34983 requested a full tablet report, sending...
I20260619 13:12:01.731884 19199 ts_manager.cc:194] Registered new tserver with Master: 48e91b8363dc4c99b2ef2b7c0f444a41 (127.0.97.3:38719)
I20260619 13:12:01.732496 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:12:01.732787 19199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34635
I20260619 13:12:01.743556 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.736960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.0.97.62:34983)
0619 13:12:01.737137 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.737141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.737696 (+ 555us) server_negotiation.cc:407] Connection header received
0619 13:12:01.737929 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.737933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.738008 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.738111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:01.738598 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.739305 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.740902 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.741432 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.741536 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.742878 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:01.742883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:01.742886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:01.742904 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:01.742947 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:01.742962 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:01.743090 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.743253 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.743300 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:12:01.747452 19199 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:12:01.754362 19199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:12:01.779763 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.767093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57172)
0619 13:12:01.769118 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.769147 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.769286 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.771476 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.771485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.771517 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:01.771817 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.771826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.774163 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.774169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.776570 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.776586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.778743 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.778753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.778969 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.779037 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.779169 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.779395 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1840,"thread_start_us":805,"threads_started":1}
I20260619 13:12:01.780038 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.767058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46185 (local address 127.0.0.1:55630)
0619 13:12:01.768022 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.768087 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.768337 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.772152 (+ 3815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.772157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.772182 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:01.772643 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.772656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.774849 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.774855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.776983 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.776997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.779098 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.779120 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.779332 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.779366 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.779427 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.779582 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":4,"thread_start_us":1670,"threads_started":1}
I20260619 13:12:01.780988 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.767688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.0.97.1:44895)
0619 13:12:01.770735 (+ 3047us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.770745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.770778 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:12:01.770881 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.770889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.771208 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.772028 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:01.772345 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.774013 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.776880 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.778502 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.780037 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.780312 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.780389 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.780595 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2839,"thread_start_us":2644,"threads_started":1}
I20260619 13:12:01.781698 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.767155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.0.97.2:46185)
0619 13:12:01.771609 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.771616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.771639 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:12:01.771728 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.771734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.771900 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.772862 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:01.773112 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.774694 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.777238 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.778862 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.780786 (+ 1924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.781020 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.781101 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.781280 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4243,"spinlock_wait_cycles":22784,"thread_start_us":4200,"threads_started":1}
I20260619 13:12:01.783488 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.770380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38719 (local address 127.0.0.1:55106)
0619 13:12:01.770937 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.770957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.771073 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.773452 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.773457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.773484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:01.773845 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.773857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.777408 (+ 3551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.777413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.779575 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.779587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.782670 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.782677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.782863 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.782897 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.783027 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.783175 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":350,"threads_started":1}
I20260619 13:12:01.783555 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.770884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.0.97.3:38719)
0619 13:12:01.771535 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.771544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.771575 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:12:01.771667 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.771675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.771900 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.774030 (+ 2130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:01.774292 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.777255 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.780775 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.782389 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.782611 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.782883 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.783068 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.783233 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":300,"threads_started":1}
I20260619 13:12:01.785342 19451 tablet_service.cc:1467] Processing CreateTablet for tablet 07896762ee314d82a75e80dfadbdecc3 (DEFAULT_TABLE table=test-table [id=420bbe98951e48d0941b1c4ea39e5517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:01.785367 19320 tablet_service.cc:1467] Processing CreateTablet for tablet 07896762ee314d82a75e80dfadbdecc3 (DEFAULT_TABLE table=test-table [id=420bbe98951e48d0941b1c4ea39e5517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:01.786932 19451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896762ee314d82a75e80dfadbdecc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:01.787448 19320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896762ee314d82a75e80dfadbdecc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:01.787259 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 07896762ee314d82a75e80dfadbdecc3 (DEFAULT_TABLE table=test-table [id=420bbe98951e48d0941b1c4ea39e5517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:01.788389 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896762ee314d82a75e80dfadbdecc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:01.798331 19385 heartbeater.cc:503] Master 127.0.97.62:34983 was elected leader, sending a full tablet report...
I20260619 13:12:01.802008 19664 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Bootstrap starting.
I20260619 13:12:01.805425 19665 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Bootstrap starting.
I20260619 13:12:01.806198 19664 tablet_bootstrap.cc:654] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:01.807662 19666 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Bootstrap starting.
I20260619 13:12:01.807695 19664 log.cc:824] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:01.809712 19665 tablet_bootstrap.cc:654] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:01.811223 19666 tablet_bootstrap.cc:654] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:01.811308 19665 log.cc:824] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:01.811597 19664 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: No bootstrap required, opened a new log
I20260619 13:12:01.811910 19664 ts_tablet_manager.cc:1397] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:12:01.812546 19666 log.cc:824] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:01.814550 19665 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: No bootstrap required, opened a new log
I20260619 13:12:01.814829 19665 ts_tablet_manager.cc:1397] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:12:01.815043 19666 tablet_bootstrap.cc:492] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: No bootstrap required, opened a new log
I20260619 13:12:01.815284 19666 ts_tablet_manager.cc:1397] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:12:01.822599 19664 raft_consensus.cc:348] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.823078 19664 raft_consensus.cc:374] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:01.822902 19666 raft_consensus.cc:348] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.823220 19664 raft_consensus.cc:729] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e91b8363dc4c99b2ef2b7c0f444a41, State: Initialized, Role: FOLLOWER
I20260619 13:12:01.823343 19666 raft_consensus.cc:374] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:01.823500 19666 raft_consensus.cc:729] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe50edbd4eb4f7ab63a4e5de7cd45e7, State: Initialized, Role: FOLLOWER
I20260619 13:12:01.823895 19664 consensus_queue.cc:260] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.824079 19666 consensus_queue.cc:260] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.825296 19516 heartbeater.cc:503] Master 127.0.97.62:34983 was elected leader, sending a full tablet report...
I20260619 13:12:01.826022 19666 ts_tablet_manager.cc:1428] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 13:12:01.826022 19665 raft_consensus.cc:348] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.826524 19665 raft_consensus.cc:374] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:01.826620 19665 raft_consensus.cc:729] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12302e22ea543e0acc03305dc444a3a, State: Initialized, Role: FOLLOWER
I20260619 13:12:01.827006 19646 heartbeater.cc:503] Master 127.0.97.62:34983 was elected leader, sending a full tablet report...
I20260619 13:12:01.827282 19665 consensus_queue.cc:260] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.827553 19664 ts_tablet_manager.cc:1428] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 13:12:01.828284 19670 raft_consensus.cc:482] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:12:01.828627 19670 raft_consensus.cc:504] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.838912 19670 leader_election.cc:288] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe50edbd4eb4f7ab63a4e5de7cd45e7 (127.0.97.2:46185), d12302e22ea543e0acc03305dc444a3a (127.0.97.1:44895)
I20260619 13:12:01.843398 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.831748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39925 (local address 127.0.97.2:46185)
0619 13:12:01.831918 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.831925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.831943 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:12:01.832017 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.832021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.832089 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.832169 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:01.832666 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.833868 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.838843 (+ 4975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.840491 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.840639 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.840773 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.842840 (+ 2067us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.843038 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:12:01.843755 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.830985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46185 (local address 127.0.97.3:39925)
0619 13:12:01.831380 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.831403 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.831534 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.832336 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.832340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.832370 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:01.832519 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.832526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.835738 (+ 3212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.835747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.838573 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.838594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.841829 (+ 3235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.841838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.842147 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.842205 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.842319 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.842527 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260619 13:12:01.844966 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896762ee314d82a75e80dfadbdecc3" candidate_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" is_pre_election: true
I20260619 13:12:01.845557 19471 raft_consensus.cc:2454] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e91b8363dc4c99b2ef2b7c0f444a41 in term 0.
I20260619 13:12:01.846334 19534 leader_election.cc:302] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e91b8363dc4c99b2ef2b7c0f444a41, cfe50edbd4eb4f7ab63a4e5de7cd45e7; no voters:
I20260619 13:12:01.846750 19672 raft_consensus.cc:2781] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:12:01.846939 19672 raft_consensus.cc:482] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:12:01.847138 19672 raft_consensus.cc:3037] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:01.847965 19665 ts_tablet_manager.cc:1428] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Time spent starting tablet: real 0.033s user 0.014s sys 0.018s
I20260619 13:12:01.850756 19672 raft_consensus.cc:504] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.851338 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.835020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48103 (local address 127.0.97.1:44895)
0619 13:12:01.835567 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.835576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.844284 (+ 8708us) server_negotiation.cc:407] Connection header received
0619 13:12:01.844606 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.844615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.844741 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.844821 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:01.845815 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.847128 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.849485 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.850425 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.850579 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.850679 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.850986 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.851119 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260619 13:12:01.851372 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.833191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.97.3:48103)
0619 13:12:01.844152 (+ 10961us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:01.844178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:01.844414 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:01.845094 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:01.845102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:01.845135 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:01.845546 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.845562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.847335 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.847340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.849280 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:01.849295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.850611 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.850617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:01.850797 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.850828 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:12:01.850947 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 13:12:01.851079 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10790,"thread_start_us":13043,"threads_started":1}
I20260619 13:12:01.852101 19672 leader_election.cc:288] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [CANDIDATE]: Term 1 election: Requested vote from peers cfe50edbd4eb4f7ab63a4e5de7cd45e7 (127.0.97.2:46185), d12302e22ea543e0acc03305dc444a3a (127.0.97.1:44895)
I20260619 13:12:01.852064 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896762ee314d82a75e80dfadbdecc3" candidate_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7"
I20260619 13:12:01.852278 19471 raft_consensus.cc:3037] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:01.852795 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896762ee314d82a75e80dfadbdecc3" candidate_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12302e22ea543e0acc03305dc444a3a" is_pre_election: true
I20260619 13:12:01.853067 19339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896762ee314d82a75e80dfadbdecc3" candidate_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12302e22ea543e0acc03305dc444a3a"
I20260619 13:12:01.853401 19340 raft_consensus.cc:2454] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e91b8363dc4c99b2ef2b7c0f444a41 in term 0.
I20260619 13:12:01.855015 19471 raft_consensus.cc:2454] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e91b8363dc4c99b2ef2b7c0f444a41 in term 1.
I20260619 13:12:01.855433 19534 leader_election.cc:302] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e91b8363dc4c99b2ef2b7c0f444a41, cfe50edbd4eb4f7ab63a4e5de7cd45e7; no voters: d12302e22ea543e0acc03305dc444a3a
I20260619 13:12:01.855773 19672 raft_consensus.cc:2781] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:01.855959 19672 raft_consensus.cc:686] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 1 LEADER]: Becoming Leader. State: Replica: 48e91b8363dc4c99b2ef2b7c0f444a41, State: Running, Role: LEADER
I20260619 13:12:01.856462 19672 consensus_queue.cc:237] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } }
I20260619 13:12:01.860832 19199 catalog_manager.cc:5495] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e91b8363dc4c99b2ef2b7c0f444a41 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e91b8363dc4c99b2ef2b7c0f444a41" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } health_report { overall_health: HEALTHY } } }
I20260619 13:12:01.889032 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.883718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.0.97.3:38719)
0619 13:12:01.883893 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.883897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.883917 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:12:01.884111 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.884115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.884195 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.884339 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:01.884660 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.885377 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.886847 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.887300 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.887400 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.888051 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:01.888561 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:01.888712 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.888793 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.888831 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:12:01.897094 19339 raft_consensus.cc:3037] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:01.897331 19471 raft_consensus.cc:1261] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 1 FOLLOWER]: Refusing update from remote peer 48e91b8363dc4c99b2ef2b7c0f444a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:01.898253 19672 consensus_queue.cc:1035] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe50edbd4eb4f7ab63a4e5de7cd45e7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:01.899798 19339 raft_consensus.cc:1261] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 1 FOLLOWER]: Refusing update from remote peer 48e91b8363dc4c99b2ef2b7c0f444a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:01.900629 19672 consensus_queue.cc:1035] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12302e22ea543e0acc03305dc444a3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:01.926972 19198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:01.927305 19198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:01.931118 19198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 266dcaaff0a743deb263de7d3b4b3251: Sending DeleteTablet for 3 replicas of tablet 07896762ee314d82a75e80dfadbdecc3
I20260619 13:12:01.932386 19451 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896762ee314d82a75e80dfadbdecc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:01 UTC) from {username='slave'} at 127.0.0.1:55630
I20260619 13:12:01.932530 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896762ee314d82a75e80dfadbdecc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:01 UTC) from {username='slave'} at 127.0.0.1:55106
I20260619 13:12:01.933221 19320 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896762ee314d82a75e80dfadbdecc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:01 UTC) from {username='slave'} at 127.0.0.1:57172
I20260619 13:12:01.934087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19258
I20260619 13:12:01.936311 19686 tablet_replica.cc:330] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: stopping tablet replica
I20260619 13:12:01.936796 19686 raft_consensus.cc:2229] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:12:01.937543 19686 raft_consensus.cc:2258] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:01.939013 19688 tablet_replica.cc:330] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: stopping tablet replica
I20260619 13:12:01.939617 19688 raft_consensus.cc:2229] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:01.940182 19688 raft_consensus.cc:2258] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:01.943830 19686 ts_tablet_manager.cc:1905] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:01.944361 19688 ts_tablet_manager.cc:1905] T 07896762ee314d82a75e80dfadbdecc3 P d12302e22ea543e0acc03305dc444a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:01.946725 19687 tablet_replica.cc:330] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: stopping tablet replica
I20260619 13:12:01.947427 19687 raft_consensus.cc:2229] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:01.947996 19687 raft_consensus.cc:2258] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:01.948807 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:01.935056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.0.97.1:44895)
0619 13:12:01.935245 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:01.935251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:01.938405 (+ 3154us) server_negotiation.cc:407] Connection header received
0619 13:12:01.938558 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:01.938571 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:01.938717 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:01.938866 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:01.939883 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.941564 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.944153 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:01.945045 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:01.945171 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:01.947012 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:01.947023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:01.947041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:01.947838 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:01.947897 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:01.947940 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:01.948044 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:01.948330 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 13:12:01.948462 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:12:01.950001 19380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:01.950695 19687 ts_tablet_manager.cc:1905] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:01.957569 19686 ts_tablet_manager.cc:1918] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:01.957943 19686 log.cc:1196] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/07896762ee314d82a75e80dfadbdecc3
I20260619 13:12:01.958602 19686 ts_tablet_manager.cc:1939] T 07896762ee314d82a75e80dfadbdecc3 P 48e91b8363dc4c99b2ef2b7c0f444a41: Deleting consensus metadata
I20260619 13:12:01.959990 19687 ts_tablet_manager.cc:1918] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:01.960196 19687 log.cc:1196] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/07896762ee314d82a75e80dfadbdecc3
I20260619 13:12:01.960776 19184 catalog_manager.cc:4841] TS 48e91b8363dc4c99b2ef2b7c0f444a41 (127.0.97.3:38719): tablet 07896762ee314d82a75e80dfadbdecc3 (table test-table [id=420bbe98951e48d0941b1c4ea39e5517]) successfully deleted
I20260619 13:12:01.961119 19687 ts_tablet_manager.cc:1939] T 07896762ee314d82a75e80dfadbdecc3 P cfe50edbd4eb4f7ab63a4e5de7cd45e7: Deleting consensus metadata
I20260619 13:12:01.963866 19183 catalog_manager.cc:4841] TS cfe50edbd4eb4f7ab63a4e5de7cd45e7 (127.0.97.2:46185): tablet 07896762ee314d82a75e80dfadbdecc3 (table test-table [id=420bbe98951e48d0941b1c4ea39e5517]) successfully deleted
I20260619 13:12:02.053923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19258
I20260619 13:12:02.072346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19389
W20260619 13:12:02.072296 19184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44895 (error 108)
I20260619 13:12:02.074450 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.073782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57184)
0619 13:12:02.074170 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.074288 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
W20260619 13:12:02.074785 19184 catalog_manager.cc:4584] TS d12302e22ea543e0acc03305dc444a3a (127.0.97.1:44895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07896762ee314d82a75e80dfadbdecc3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
I20260619 13:12:02.081072 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.073939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.97.2:46185)
0619 13:12:02.074142 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:02.074146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:02.074166 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:12:02.074227 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:02.074231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:02.074343 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:02.074511 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:02.075028 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:02.075896 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:02.077302 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:02.077996 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:02.078153 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:02.079429 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:02.079440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:02.079456 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:02.080144 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:02.080219 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:02.080243 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:02.080422 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:02.080568 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 13:12:02.080689 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:12:02.081863 19511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:02.128582 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.127940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57198)
0619 13:12:02.128227 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.128350 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 13:12:02.129997 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.129504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57202)
0619 13:12:02.129712 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.129811 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:12:02.176577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19389
I20260619 13:12:02.191103 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.190435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57216)
0619 13:12:02.190710 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.190833 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 13:12:02.192481 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.191966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57224)
0619 13:12:02.192172 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.192300 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 13:12:02.194643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19519
I20260619 13:12:02.202505 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.195428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.97.3:38719)
0619 13:12:02.195601 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:02.195607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:02.195655 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:12:02.195874 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:02.195880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:02.195977 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:02.196095 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:02.196567 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:02.197394 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:02.198830 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:02.199355 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:02.199508 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:02.200714 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:02.200725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:02.200740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:02.201401 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:02.201458 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:02.201479 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:02.201684 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:02.201904 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 13:12:02.202021 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:12:02.203325 19641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:02.273415 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.272804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57234)
0619 13:12:02.273114 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.273212 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 13:12:02.274559 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.274192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57246)
0619 13:12:02.274379 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.274453 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 13:12:02.330708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19519
I20260619 13:12:02.352721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19168
I20260619 13:12:02.353926 19229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:02.466696 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:02.465769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44895 (local address 127.0.0.1:57258)
0619 13:12:02.465991 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:02.466406 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 13:12:02.466709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19168
2026-06-19T13:12:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2802 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:02 dist-test-slave-q11v krb5kdc[19697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:02 dist-test-slave-q11v krb5kdc[19697](info): set up 2 sockets
Jun 19 13:12:02 dist-test-slave-q11v krb5kdc[19697](info): commencing operation
krb5kdc: starting...
W20260619 13:12:03.585937 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:03 dist-test-slave-q11v krb5kdc[19697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:03.840384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39449
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:04.179131 19710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:04.179435 19710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:04.179602 19710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:04.186983 19710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:04.187103 19710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:04.187196 19710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:04.187302 19710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:04.187371 19710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:04.197422 19710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:04.198794 19710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:04.200167 19710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:04.212255 19715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:04.212266 19716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:04.212345 19718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:04.213584 19710 server_base.cc:1034] running on GCE node
I20260619 13:12:04.214495 19710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:04.216023 19710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:04.217269 19710 hybrid_clock.cc:648] HybridClock initialized: now 1781874724217211 us; error 57 us; skew 500 ppm
I20260619 13:12:04.217758 19710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:04 dist-test-slave-q11v krb5kdc[19697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:04.226754 19710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:04.228565 19710 webserver.cc:460] Webserver started at http://127.0.97.62:45757/ using document root <none> and password file <none>
I20260619 13:12:04.229296 19710 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:04.229404 19710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:04.229761 19710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:04.232834 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "9244b3dc8e74442abc72e052fb383fcb"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "960a192008d3e17e8f14be6698d7074f"
server_key_iv: "19b298f98f4e53103dc154499caebd61"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.233582 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "9244b3dc8e74442abc72e052fb383fcb"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "960a192008d3e17e8f14be6698d7074f"
server_key_iv: "19b298f98f4e53103dc154499caebd61"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.239120 19710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:12:04.242738 19725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:04.244105 19710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:12:04.244306 19710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "9244b3dc8e74442abc72e052fb383fcb"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "960a192008d3e17e8f14be6698d7074f"
server_key_iv: "19b298f98f4e53103dc154499caebd61"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.244554 19710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:04.272305 19710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:04.277174 19710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:04.277486 19710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:04.297343 19710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39449
I20260619 13:12:04.297361 19776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39449 every 8 connection(s)
I20260619 13:12:04.298879 19710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:04.302790 19777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:04.305299 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19710
I20260619 13:12:04.305500 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:04.305977 388 external_mini_cluster.cc:1428] Setting key bc20330a22f9cb54a53e944cb2fd2d65
I20260619 13:12:04.316393 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: Bootstrap starting.
I20260619 13:12:04.320089 19777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:04.321131 19777 log.cc:824] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:12:04 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:04.324783 19777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: No bootstrap required, opened a new log
I20260619 13:12:04.332111 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:04.307980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.0.97.62:39449)
0619 13:12:04.308862 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:04.308878 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:04.308931 (+ 53us) server_negotiation.cc:407] Connection header received
0619 13:12:04.311397 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:04.311430 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:04.311994 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:04.312495 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:04.313857 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.315851 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.317270 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.318395 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.318595 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:04.323045 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:04.323057 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:04.323078 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:04.323130 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:04.327231 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:04.327999 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:04.328008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:04.328018 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:04.328254 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:04.328654 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:04.328658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:04.328662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:04.329392 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:04.329746 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:04.330121 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 13:12:04.330628 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":353,"threads_started":1}
I20260619 13:12:04.335554 19777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } }
I20260619 13:12:04.335888 19777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:04.335963 19777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9244b3dc8e74442abc72e052fb383fcb, State: Initialized, Role: FOLLOWER
I20260619 13:12:04.336449 19777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } }
I20260619 13:12:04.336651 19777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:04.336742 19777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:04.336861 19777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:04.339084 19777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } }
I20260619 13:12:04.339536 19777 leader_election.cc:302] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9244b3dc8e74442abc72e052fb383fcb; no voters:
I20260619 13:12:04.340204 19777 leader_election.cc:288] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:04.340353 19782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:04.341317 19782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [term 1 LEADER]: Becoming Leader. State: Replica: 9244b3dc8e74442abc72e052fb383fcb, State: Running, Role: LEADER
I20260619 13:12:04.341898 19782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } }
I20260619 13:12:04.342391 19777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:04.344512 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9244b3dc8e74442abc72e052fb383fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } } }
I20260619 13:12:04.344940 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:04.345463 19784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9244b3dc8e74442abc72e052fb383fcb. Latest consensus state: current_term: 1 leader_uuid: "9244b3dc8e74442abc72e052fb383fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9244b3dc8e74442abc72e052fb383fcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39449 } } }
I20260619 13:12:04.345767 19784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:04.345903 19791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:04.352119 19791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:04.360603 19791 catalog_manager.cc:1347] Generated new cluster ID: 4ab0a8db751a4ebcb166789a71b6bcf9
I20260619 13:12:04.360743 19791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:04.373179 19791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:04.375067 19791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:04.387560 19791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: Generated new TSK 0
I20260619 13:12:04.388471 19791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:12:04.448586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:04.788172 19803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:04.788473 19803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:04.788643 19803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:04.796114 19803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:04.796229 19803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:04.796406 19803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:12:04.806391 19803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:04.807704 19803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:04.809062 19803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:04.823303 19808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:04.823367 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:04.823345 19809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:04.824378 19803 server_base.cc:1034] running on GCE node
I20260619 13:12:04.825062 19803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:04.826088 19803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:04.827317 19803 hybrid_clock.cc:648] HybridClock initialized: now 1781874724827280 us; error 61 us; skew 500 ppm
I20260619 13:12:04.827651 19803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:04 dist-test-slave-q11v krb5kdc[19697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:04.832842 19803 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:12:04.834542 19803 webserver.cc:460] Webserver started at http://127.0.97.1:42135/ using document root <none> and password file <none>
I20260619 13:12:04.835268 19803 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:04.835424 19803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:04.835741 19803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:04.838738 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "bb27ccd9c8494758a432406069e44c2e"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "b09f440b3655bc7075be054bb3924469"
server_key_iv: "1cc4464e86694ec0e50909ee65562c69"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.839553 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bb27ccd9c8494758a432406069e44c2e"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "b09f440b3655bc7075be054bb3924469"
server_key_iv: "1cc4464e86694ec0e50909ee65562c69"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.845325 19803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:12:04.849030 19818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:04.850359 19803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:04.850530 19803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "bb27ccd9c8494758a432406069e44c2e"
format_stamp: "Formatted at 2026-06-19 13:12:04 on dist-test-slave-q11v"
server_key: "b09f440b3655bc7075be054bb3924469"
server_key_iv: "1cc4464e86694ec0e50909ee65562c69"
server_key_version: "encryptionkey@0"
I20260619 13:12:04.850699 19803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:04.882854 19803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:04.887801 19803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:04.888101 19803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:04.889097 19803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:04.890651 19803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:04.890726 19803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:04.890901 19803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:04.890961 19803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:04.924415 19803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43291
I20260619 13:12:04.924465 19930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43291 every 8 connection(s)
I20260619 13:12:04.925777 19803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:12:04.934410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19803
I20260619 13:12:04.934581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:12:04.935091 388 external_mini_cluster.cc:1428] Setting key 9ab56e211c7f965a5f942f6199b86e43
Jun 19 13:12:04 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:04.948414 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:04.928498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40949 (local address 127.0.97.62:39449)
0619 13:12:04.929016 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:04.929023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:04.930066 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 13:12:04.931918 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:04.931922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:04.932003 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:04.932105 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:04.933882 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.935030 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.936855 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.937431 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.937564 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:04.941267 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:04.941276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:04.941284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:04.941330 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:04.944969 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:04.945782 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:04.945790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:04.945795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:04.945929 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:04.946431 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:04.946434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:04.946436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:04.946687 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:04.946795 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:04.947706 (+ 911us) server_negotiation.cc:299] Negotiation successful
0619 13:12:04.948034 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
I20260619 13:12:04.949751 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:04.929152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39449 (local address 127.0.97.1:40949)
0619 13:12:04.929902 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:04.929947 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:04.931622 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:04.932320 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:04.932328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:04.932956 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:04.933685 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:04.933710 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.935233 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.935237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:04.936675 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:04.936687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:04.937699 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:04.937705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:04.937910 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:04.938639 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:04.938663 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:04.941041 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:04.945141 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:04.945148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:04.945161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:04.945643 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:04.946039 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:04.946045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:04.946050 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:04.946314 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:04.946824 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:04.946833 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:04.947300 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 13:12:04.947605 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 13:12:04.948227 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":298,"threads_started":1}
I20260619 13:12:04.951321 19931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39449
I20260619 13:12:04.951640 19931 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:04.952387 19931 heartbeater.cc:511] Master 127.0.97.62:39449 requested a full tablet report, sending...
I20260619 13:12:04.954538 19742 ts_manager.cc:194] Registered new tserver with Master: bb27ccd9c8494758a432406069e44c2e (127.0.97.1:43291)
I20260619 13:12:04.955999 19742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40949
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:12:04.991448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:05.326252 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:05.326579 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:05.326742 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:05.334139 19937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:05.334287 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:05.334457 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:12:05.344297 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:05.345677 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:05.347043 19937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:05.360388 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:05.360417 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:05.360445 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:05.361672 19937 server_base.cc:1034] running on GCE node
I20260619 13:12:05.362321 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:05.363293 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:05.364523 19937 hybrid_clock.cc:648] HybridClock initialized: now 1781874725364477 us; error 44 us; skew 500 ppm
I20260619 13:12:05.364847 19937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:05 dist-test-slave-q11v krb5kdc[19697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:05.368120 19937 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:12:05.369563 19937 webserver.cc:460] Webserver started at http://127.0.97.2:33983/ using document root <none> and password file <none>
I20260619 13:12:05.370333 19937 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:05.370478 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:05.370787 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:05.373703 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "1324db7adbe64f36b07f14b275fdab17"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "f4084ee5013125fe9c0f4581f36bbb3b"
server_key_iv: "2f9090bdf16ff33de7c87f7f48305ca1"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.374567 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1324db7adbe64f36b07f14b275fdab17"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "f4084ee5013125fe9c0f4581f36bbb3b"
server_key_iv: "2f9090bdf16ff33de7c87f7f48305ca1"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.379935 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:12:05.383484 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:05.384840 19937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:12:05.385035 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "1324db7adbe64f36b07f14b275fdab17"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "f4084ee5013125fe9c0f4581f36bbb3b"
server_key_iv: "2f9090bdf16ff33de7c87f7f48305ca1"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.385244 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:05.401638 19937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:05.406613 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:05.406956 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:05.408011 19937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:05.409644 19937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:05.409757 19937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 13:12:05.409931 19937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:05.410010 19937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:05.446489 19937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43227
I20260619 13:12:05.446632 20064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43227 every 8 connection(s)
I20260619 13:12:05.447988 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:12:05.457711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 19937
I20260619 13:12:05.457917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:12:05.458463 388 external_mini_cluster.cc:1428] Setting key de2264cf2b1b0fd4b6256fabd9419111
Jun 19 13:12:05 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:05.471601 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:05.450899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33911 (local address 127.0.97.62:39449)
0619 13:12:05.451352 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:05.451359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:05.452304 (+ 945us) server_negotiation.cc:407] Connection header received
0619 13:12:05.454274 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:05.454280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:05.454375 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:05.454507 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:05.456094 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:05.456820 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:05.459176 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:05.459801 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:05.459951 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:05.464037 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:05.464046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:05.464054 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:05.464103 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:05.467998 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:05.468805 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:05.468814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:05.468828 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:05.468991 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:05.469506 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:05.469514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:05.469520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:05.469951 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:05.470067 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:05.471119 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0619 13:12:05.471308 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
I20260619 13:12:05.472668 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:05.451448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39449 (local address 127.0.97.2:33911)
0619 13:12:05.452148 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:05.452191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:05.454006 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:05.454744 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:05.454754 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:05.455345 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:05.455917 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:05.455936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:05.456989 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:05.456993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:05.459007 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:05.459023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:05.460038 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:05.460045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:05.460288 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:05.461001 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:05.461031 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:05.463783 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:05.468168 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:05.468175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:05.468189 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:05.468660 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:05.469133 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:05.469137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:05.469140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:05.469372 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:05.470100 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:05.470111 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:05.470614 (+ 503us) client_negotiation.cc:769] Sending connection context
0619 13:12:05.470973 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 13:12:05.471465 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":284,"threads_started":1}
I20260619 13:12:05.474251 20065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39449
I20260619 13:12:05.474581 20065 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:05.475469 20065 heartbeater.cc:511] Master 127.0.97.62:39449 requested a full tablet report, sending...
I20260619 13:12:05.477119 19742 ts_manager.cc:194] Registered new tserver with Master: 1324db7adbe64f36b07f14b275fdab17 (127.0.97.2:43227)
I20260619 13:12:05.478081 19742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33911
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:12:05.517320 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:05.855757 20071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:05.856071 20071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:05.856174 20071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:05.863798 20071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:05.863930 20071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:05.864056 20071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:12:05.874389 20071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:05.875726 20071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:05.877079 20071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:05.891279 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:05.891248 20077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:05.891824 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:05.892127 20071 server_base.cc:1034] running on GCE node
I20260619 13:12:05.892824 20071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:05.893985 20071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:05.895233 20071 hybrid_clock.cc:648] HybridClock initialized: now 1781874725895169 us; error 59 us; skew 500 ppm
I20260619 13:12:05.895622 20071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:05 dist-test-slave-q11v krb5kdc[19697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:05.899010 20071 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:12:05.900667 20071 webserver.cc:460] Webserver started at http://127.0.97.3:46859/ using document root <none> and password file <none>
I20260619 13:12:05.901475 20071 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:05.901678 20071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:05.902024 20071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:05.905028 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "66d7593e9f104b1f88090d45209476bd"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "aeaf66bf208d85bdf058187892ff0369"
server_key_iv: "ba979aff22ea0e4c81c0dbda04725180"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.905948 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "66d7593e9f104b1f88090d45209476bd"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "aeaf66bf208d85bdf058187892ff0369"
server_key_iv: "ba979aff22ea0e4c81c0dbda04725180"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.911262 20071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:12:05.914949 20086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:05.916410 20071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:05.916553 20071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "66d7593e9f104b1f88090d45209476bd"
format_stamp: "Formatted at 2026-06-19 13:12:05 on dist-test-slave-q11v"
server_key: "aeaf66bf208d85bdf058187892ff0369"
server_key_iv: "ba979aff22ea0e4c81c0dbda04725180"
server_key_version: "encryptionkey@0"
I20260619 13:12:05.916740 20071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:05.946472 20071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:05.951287 20071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:05.951532 20071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:05.952540 20071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:05.953980 20071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:05.954056 20071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:05.954137 20071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:05.954180 20071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:05.959149 19931 heartbeater.cc:503] Master 127.0.97.62:39449 was elected leader, sending a full tablet report...
I20260619 13:12:05.989629 20071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34619
I20260619 13:12:05.989691 20198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34619 every 8 connection(s)
I20260619 13:12:05.990950 20071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:12:05.993208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20071
I20260619 13:12:05.993391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:12:05.994145 388 external_mini_cluster.cc:1428] Setting key 84854c950aa7af97da723252b8d52943
Jun 19 13:12:06 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:06.013559 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:05.994480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39449 (local address 127.0.97.3:49839)
0619 13:12:05.995349 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:05.995400 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:05.997106 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:05.997930 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:05.997940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:05.998457 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:05.999028 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:05.999057 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.000098 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.000102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.001352 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.001367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.002147 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.002152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.002354 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.002966 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:06.002986 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:06.005159 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:06.008256 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:06.008266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:06.008282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:06.009094 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:06.009627 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:06.009632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:06.009635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:06.009892 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:06.010667 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:06.010675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:06.011180 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.011555 (+ 375us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.012139 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":371,"threads_started":1}
I20260619 13:12:06.014735 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:05.993703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49839 (local address 127.0.97.62:39449)
0619 13:12:05.994168 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:05.994173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:05.997421 (+ 3248us) server_negotiation.cc:407] Connection header received
0619 13:12:05.997474 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:05.997477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:05.997551 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:05.997702 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:05.999214 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:05.999969 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.001491 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.001964 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.002061 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.005354 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:06.005359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:06.005362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:06.005393 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:06.008115 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.009263 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.009269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.009272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.009461 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.010058 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.010063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.010067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.010499 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:06.010682 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.014112 (+ 3430us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.014353 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260619 13:12:06.018491 20199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39449
I20260619 13:12:06.018890 20199 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:06.019698 20199 heartbeater.cc:511] Master 127.0.97.62:39449 requested a full tablet report, sending...
I20260619 13:12:06.021390 19742 ts_manager.cc:194] Registered new tserver with Master: 66d7593e9f104b1f88090d45209476bd (127.0.97.3:34619)
I20260619 13:12:06.022562 19742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49839
I20260619 13:12:06.022658 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:12:06.039492 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.026886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.0.97.62:39449)
0619 13:12:06.027053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.027057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.027552 (+ 495us) server_negotiation.cc:407] Connection header received
0619 13:12:06.027648 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.027652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.027716 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.027803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:06.029274 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.030264 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.031615 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.032052 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.032157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.034642 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:06.034647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:06.034650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:06.034679 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:06.037098 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.037791 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.037795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.037797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.037917 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.038262 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.038266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.038268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.038593 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:06.038784 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.039057 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.039229 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:12:06.043438 19742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:12:06.050659 19742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:12:06.074741 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.063678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43291 (local address 127.0.0.1:50976)
0619 13:12:06.064753 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:06.064813 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:06.065033 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:06.066497 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:06.066506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:06.066538 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:06.066923 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.066936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.069200 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.069206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.071780 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.071794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.073767 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.073774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.073968 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.074030 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.074137 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.074338 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":2093,"threads_started":1}
I20260619 13:12:06.076766 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.063929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.0.97.1:43291)
0619 13:12:06.065805 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.065817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.065848 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:12:06.065952 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.065959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.066242 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.067142 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:06.067385 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.069051 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.071991 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.073530 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.075923 (+ 2393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.076161 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.076236 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.076416 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1701,"thread_start_us":1489,"threads_started":1}
I20260619 13:12:06.077399 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.063904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34619 (local address 127.0.0.1:39640)
0619 13:12:06.066065 (+ 2161us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:06.066082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:06.066189 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:06.067796 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:06.067800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:06.067824 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:06.068223 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.068235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.071407 (+ 3172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.071412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.074652 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.074666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.076535 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.076542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.076755 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.076789 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.076877 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.077036 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2024,"thread_start_us":1030,"threads_started":1}
I20260619 13:12:06.077661 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.064631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.3:34619)
0619 13:12:06.066760 (+ 2129us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.066768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.066803 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:12:06.066904 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.066911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.067175 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.067515 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:06.068422 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.071236 (+ 2814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.074855 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.076335 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.076465 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.076685 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.077112 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.077286 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1954,"thread_start_us":1781,"threads_started":1}
I20260619 13:12:06.081028 19866 tablet_service.cc:1467] Processing CreateTablet for tablet ee4010e9e49145969950c7e5179e3a53 (DEFAULT_TABLE table=test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:06.081526 20134 tablet_service.cc:1467] Processing CreateTablet for tablet ee4010e9e49145969950c7e5179e3a53 (DEFAULT_TABLE table=test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:06.082667 19866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4010e9e49145969950c7e5179e3a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:06.083088 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.065959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43227 (local address 127.0.0.1:44316)
0619 13:12:06.075375 (+ 9416us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:06.075397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:06.075503 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:06.076791 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:06.076796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:06.076825 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:06.077001 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.077011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.079461 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.079467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.080754 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.080767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.082290 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.082296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.082497 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.082534 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.082620 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.082781 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9280,"thread_start_us":9577,"threads_started":1}
I20260619 13:12:06.083261 20134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4010e9e49145969950c7e5179e3a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:06.083814 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.066685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.0.97.2:43227)
0619 13:12:06.076152 (+ 9467us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.076158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.076231 (+ 73us) server_negotiation.cc:407] Connection header received
0619 13:12:06.076329 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.076334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.076590 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.076815 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:06.077851 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.079300 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.080914 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.081971 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.082118 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.082317 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.083447 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.083585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9312,"thread_start_us":9170,"threads_started":1}
I20260619 13:12:06.087380 20000 tablet_service.cc:1467] Processing CreateTablet for tablet ee4010e9e49145969950c7e5179e3a53 (DEFAULT_TABLE table=test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:06.088801 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4010e9e49145969950c7e5179e3a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:06.097455 20218 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Bootstrap starting.
I20260619 13:12:06.098501 20219 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Bootstrap starting.
I20260619 13:12:06.101433 20218 tablet_bootstrap.cc:654] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:06.102159 20219 tablet_bootstrap.cc:654] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:06.102807 20218 log.cc:824] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:06.103433 20219 log.cc:824] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:06.103543 20220 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Bootstrap starting.
I20260619 13:12:06.106820 20220 tablet_bootstrap.cc:654] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:06.107635 20218 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: No bootstrap required, opened a new log
I20260619 13:12:06.107632 20219 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: No bootstrap required, opened a new log
I20260619 13:12:06.107908 20219 ts_tablet_manager.cc:1397] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:12:06.107910 20218 ts_tablet_manager.cc:1397] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 13:12:06.107995 20220 log.cc:824] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:06.110795 20220 tablet_bootstrap.cc:492] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: No bootstrap required, opened a new log
I20260619 13:12:06.111070 20220 ts_tablet_manager.cc:1397] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:12:06.118430 20220 raft_consensus.cc:348] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.118829 20220 raft_consensus.cc:374] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:06.118968 20220 raft_consensus.cc:729] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1324db7adbe64f36b07f14b275fdab17, State: Initialized, Role: FOLLOWER
I20260619 13:12:06.118911 20218 raft_consensus.cc:348] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.118911 20219 raft_consensus.cc:348] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.119369 20219 raft_consensus.cc:374] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:06.119458 20218 raft_consensus.cc:374] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:06.119479 20219 raft_consensus.cc:729] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d7593e9f104b1f88090d45209476bd, State: Initialized, Role: FOLLOWER
I20260619 13:12:06.119562 20218 raft_consensus.cc:729] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb27ccd9c8494758a432406069e44c2e, State: Initialized, Role: FOLLOWER
I20260619 13:12:06.119635 20220 consensus_queue.cc:260] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.120112 20219 consensus_queue.cc:260] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.120110 20218 consensus_queue.cc:260] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.121095 20065 heartbeater.cc:503] Master 127.0.97.62:39449 was elected leader, sending a full tablet report...
I20260619 13:12:06.121809 20220 ts_tablet_manager.cc:1428] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 13:12:06.122867 20199 heartbeater.cc:503] Master 127.0.97.62:39449 was elected leader, sending a full tablet report...
I20260619 13:12:06.123502 20219 ts_tablet_manager.cc:1428] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 13:12:06.125363 20218 ts_tablet_manager.cc:1428] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260619 13:12:06.178385 19932 tablet.cc:2369] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:12:06.200460 20066 tablet.cc:2369] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:06.235724 20225 raft_consensus.cc:482] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:12:06.236040 20225 raft_consensus.cc:504] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.237883 20225 leader_election.cc:288] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d7593e9f104b1f88090d45209476bd (127.0.97.3:34619), 1324db7adbe64f36b07f14b275fdab17 (127.0.97.2:43227)
W20260619 13:12:06.242619 20200 tablet.cc:2369] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:06.245694 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.238309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42907 (local address 127.0.97.3:34619)
0619 13:12:06.238527 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.238532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.239017 (+ 485us) server_negotiation.cc:407] Connection header received
0619 13:12:06.239287 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.239296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.239386 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.239450 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:06.240169 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.241329 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.243867 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.244631 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.244788 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.244902 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.245240 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.245365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 13:12:06.245740 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.239264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48775 (local address 127.0.97.2:43227)
0619 13:12:06.239476 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.239484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.239760 (+ 276us) server_negotiation.cc:407] Connection header received
0619 13:12:06.239999 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.240011 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.240067 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.240167 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:06.240572 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.241578 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.243855 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.244629 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.244777 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.244924 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.245266 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.245439 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:12:06.245894 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.239013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43227 (local address 127.0.97.1:48775)
0619 13:12:06.239642 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:06.239657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:06.239849 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:06.240170 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:06.240172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:06.240190 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:06.240426 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.240433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.241764 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.241769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.243698 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.243712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.244951 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.244959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.245112 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.245135 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.245239 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.245367 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":9,"thread_start_us":272,"threads_started":1}
I20260619 13:12:06.245927 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.238188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34619 (local address 127.0.97.1:42907)
0619 13:12:06.238862 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:06.238883 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:06.239055 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:06.239647 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:06.239650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:06.239679 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:06.240029 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.240038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.241467 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.241470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.243711 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:06.243724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.244870 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.244876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:06.245073 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.245104 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:12:06.245213 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:12:06.245352 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":67,"spinlock_wait_cycles":20352,"thread_start_us":346,"threads_started":1}
I20260619 13:12:06.247157 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4010e9e49145969950c7e5179e3a53" candidate_uuid: "bb27ccd9c8494758a432406069e44c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1324db7adbe64f36b07f14b275fdab17" is_pre_election: true
I20260619 13:12:06.247157 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4010e9e49145969950c7e5179e3a53" candidate_uuid: "bb27ccd9c8494758a432406069e44c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d7593e9f104b1f88090d45209476bd" is_pre_election: true
I20260619 13:12:06.247750 20020 raft_consensus.cc:2454] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb27ccd9c8494758a432406069e44c2e in term 0.
I20260619 13:12:06.247784 20154 raft_consensus.cc:2454] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb27ccd9c8494758a432406069e44c2e in term 0.
I20260619 13:12:06.248467 19821 leader_election.cc:302] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d7593e9f104b1f88090d45209476bd, bb27ccd9c8494758a432406069e44c2e; no voters:
I20260619 13:12:06.248862 20225 raft_consensus.cc:2781] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:12:06.249074 20225 raft_consensus.cc:482] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:12:06.249204 20225 raft_consensus.cc:3037] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:06.251487 20225 raft_consensus.cc:504] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.252128 20225 leader_election.cc:288] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [CANDIDATE]: Term 1 election: Requested vote from peers 66d7593e9f104b1f88090d45209476bd (127.0.97.3:34619), 1324db7adbe64f36b07f14b275fdab17 (127.0.97.2:43227)
I20260619 13:12:06.252746 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4010e9e49145969950c7e5179e3a53" candidate_uuid: "bb27ccd9c8494758a432406069e44c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d7593e9f104b1f88090d45209476bd"
I20260619 13:12:06.252803 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4010e9e49145969950c7e5179e3a53" candidate_uuid: "bb27ccd9c8494758a432406069e44c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1324db7adbe64f36b07f14b275fdab17"
I20260619 13:12:06.252985 20154 raft_consensus.cc:3037] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:06.253017 20020 raft_consensus.cc:3037] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:06.256377 20020 raft_consensus.cc:2454] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb27ccd9c8494758a432406069e44c2e in term 1.
I20260619 13:12:06.256395 20154 raft_consensus.cc:2454] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb27ccd9c8494758a432406069e44c2e in term 1.
I20260619 13:12:06.256834 19821 leader_election.cc:302] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d7593e9f104b1f88090d45209476bd, bb27ccd9c8494758a432406069e44c2e; no voters:
I20260619 13:12:06.257129 20225 raft_consensus.cc:2781] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:06.257576 20225 raft_consensus.cc:686] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 1 LEADER]: Becoming Leader. State: Replica: bb27ccd9c8494758a432406069e44c2e, State: Running, Role: LEADER
I20260619 13:12:06.258140 20225 consensus_queue.cc:237] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } }
I20260619 13:12:06.262339 19741 catalog_manager.cc:5495] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb27ccd9c8494758a432406069e44c2e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bb27ccd9c8494758a432406069e44c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb27ccd9c8494758a432406069e44c2e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 } health_report { overall_health: UNKNOWN } } }
I20260619 13:12:06.315376 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.309558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.0.97.1:43291)
0619 13:12:06.309796 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.309802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.309823 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:12:06.309928 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.309933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.310030 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.310188 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:06.310578 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.311698 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.313168 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.313571 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.313733 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.314267 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:06.314834 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:06.314956 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.315070 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.315140 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:12:06.324594 20154 raft_consensus.cc:1261] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 1 FOLLOWER]: Refusing update from remote peer bb27ccd9c8494758a432406069e44c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:06.324602 20020 raft_consensus.cc:1261] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 1 FOLLOWER]: Refusing update from remote peer bb27ccd9c8494758a432406069e44c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:06.325665 20229 consensus_queue.cc:1035] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1324db7adbe64f36b07f14b275fdab17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:06.326131 20225 consensus_queue.cc:1035] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d7593e9f104b1f88090d45209476bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:06.341701 20233 mvcc.cc:205] Tried to move back new op lower bound from 7298558879013285888 to 7298558878756925440. Current Snapshot: MvccSnapshot[applied={T|T < 7298558879013285888}]
I20260619 13:12:06.354084 19741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:06.354406 19741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:06.358237 19741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9244b3dc8e74442abc72e052fb383fcb: Sending DeleteTablet for 3 replicas of tablet ee4010e9e49145969950c7e5179e3a53
I20260619 13:12:06.359411 19866 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4010e9e49145969950c7e5179e3a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50976
I20260619 13:12:06.359579 20134 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4010e9e49145969950c7e5179e3a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39640
I20260619 13:12:06.359817 20000 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4010e9e49145969950c7e5179e3a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44316
I20260619 13:12:06.361574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19803
I20260619 13:12:06.389142 20245 tablet_replica.cc:330] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: stopping tablet replica
I20260619 13:12:06.389873 20245 raft_consensus.cc:2229] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:06.390596 20245 raft_consensus.cc:2258] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:06.391717 20243 tablet_replica.cc:330] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: stopping tablet replica
I20260619 13:12:06.392434 20244 tablet_replica.cc:330] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: stopping tablet replica
I20260619 13:12:06.392844 20244 raft_consensus.cc:2229] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:06.393352 20244 raft_consensus.cc:2258] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:06.395617 20245 ts_tablet_manager.cc:1905] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:06.395617 20244 ts_tablet_manager.cc:1905] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:06.395789 20243 raft_consensus.cc:2229] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:12:06.396488 20243 raft_consensus.cc:2258] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:06.398651 20243 ts_tablet_manager.cc:1905] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:06.402979 20245 ts_tablet_manager.cc:1918] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:06.402980 20244 ts_tablet_manager.cc:1918] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:06.403220 20244 log.cc:1196] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/ee4010e9e49145969950c7e5179e3a53
I20260619 13:12:06.403224 20245 log.cc:1196] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/ee4010e9e49145969950c7e5179e3a53
I20260619 13:12:06.403911 20245 ts_tablet_manager.cc:1939] T ee4010e9e49145969950c7e5179e3a53 P 1324db7adbe64f36b07f14b275fdab17: Deleting consensus metadata
I20260619 13:12:06.404250 20244 ts_tablet_manager.cc:1939] T ee4010e9e49145969950c7e5179e3a53 P 66d7593e9f104b1f88090d45209476bd: Deleting consensus metadata
I20260619 13:12:06.406054 19728 catalog_manager.cc:4841] TS 66d7593e9f104b1f88090d45209476bd (127.0.97.3:34619): tablet ee4010e9e49145969950c7e5179e3a53 (table test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]) successfully deleted
I20260619 13:12:06.406426 19728 catalog_manager.cc:4841] TS 1324db7adbe64f36b07f14b275fdab17 (127.0.97.2:43227): tablet ee4010e9e49145969950c7e5179e3a53 (table test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]) successfully deleted
Jun 19 13:12:06 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:12:06.410409 20243 ts_tablet_manager.cc:1918] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:06.410619 20243 log.cc:1196] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/ee4010e9e49145969950c7e5179e3a53
I20260619 13:12:06.411195 20243 ts_tablet_manager.cc:1939] T ee4010e9e49145969950c7e5179e3a53 P bb27ccd9c8494758a432406069e44c2e: Deleting consensus metadata
I20260619 13:12:06.413285 19727 catalog_manager.cc:4841] TS bb27ccd9c8494758a432406069e44c2e (127.0.97.1:43291): tablet ee4010e9e49145969950c7e5179e3a53 (table test-table [id=e7490e4cd20d4ad7b5d47d8937c8e4dd]) successfully deleted
I20260619 13:12:06.416478 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.362893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.0.97.1:43291)
0619 13:12:06.363155 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.363165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.386333 (+ 23168us) server_negotiation.cc:407] Connection header received
0619 13:12:06.386876 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.386883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.386993 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.387078 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:06.391944 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.393174 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.401768 (+ 8594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.402504 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.402622 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.408798 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:06.408814 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:06.408833 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:06.408890 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:06.413267 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.414242 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.414249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.414252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.414401 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.414842 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.414850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.414852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.415231 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:06.415559 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.415877 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.416122 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:12:06.417258 19926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:06.518991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19803
I20260619 13:12:06.538170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19937
Jun 19 13:12:06 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:12:06.553941 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.539066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.0.97.2:43227)
0619 13:12:06.539256 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.539264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.539288 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:12:06.539358 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.539363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.539467 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.539582 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:06.540886 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.541689 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.543142 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.543539 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.543672 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.547950 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:06.547965 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:06.547980 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:06.548014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:06.551023 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.551801 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.551809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.551811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.551934 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.552328 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.552335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.552338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.552730 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:06.553044 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.553343 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.553560 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:12:06.554654 20060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:06.645967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19937
I20260619 13:12:06.662943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20071
Jun 19 13:12:06 dist-test-slave-q11v krb5kdc[19697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:12:06.677955 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:06.663692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.0.97.3:34619)
0619 13:12:06.663897 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:06.663905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:06.663929 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:12:06.664140 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:06.664145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:06.664221 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:06.664329 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:06.665641 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.666359 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.667689 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:06.668097 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:06.668234 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:06.672474 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:06.672491 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:06.672507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:06.672547 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:06.675135 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.675874 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.675881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.675884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.676023 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:06.676402 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:06.676409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:06.676412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:06.676766 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:06.677073 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:06.677371 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:12:06.677619 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:12:06.678630 20194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:06.769140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20071
I20260619 13:12:06.786106 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 19710
I20260619 13:12:06.787195 19772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:06.897912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 19710
2026-06-19T13:12:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4429 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T13:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:06Z Disabled control of system clock
I20260619 13:12:06.974097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34093
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:12:07.308736 20254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:07.309046 20254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:07.309211 20254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:07.316484 20254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:07.316648 20254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:07.316723 20254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:07.316823 20254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:07.316936 20254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:07.329654 20254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:07.331024 20254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:07.332760 20254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:07.345232 20259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:07.345198 20260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:07.345171 20262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:07.346357 20254 server_base.cc:1034] running on GCE node
I20260619 13:12:07.347105 20254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:07.348599 20254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:07.349792 20254 hybrid_clock.cc:648] HybridClock initialized: now 1781874727349740 us; error 56 us; skew 500 ppm
I20260619 13:12:07.350102 20254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:07.352290 20254 webserver.cc:460] Webserver started at http://127.0.97.62:43483/ using document root <none> and password file <none>
I20260619 13:12:07.352931 20254 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:07.353019 20254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:07.353276 20254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:07.356174 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "acf8bbeea57544c3861a7bdb1811925e"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "868844372d6735beaea6804697614569"
server_key_iv: "c84e51b169e42934deebe38d51bb0515"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.356977 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "acf8bbeea57544c3861a7bdb1811925e"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "868844372d6735beaea6804697614569"
server_key_iv: "c84e51b169e42934deebe38d51bb0515"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.362074 20254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:12:07.365494 20268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:07.366961 20254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:07.367141 20254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "acf8bbeea57544c3861a7bdb1811925e"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "868844372d6735beaea6804697614569"
server_key_iv: "c84e51b169e42934deebe38d51bb0515"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.367333 20254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:07.394295 20254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:07.395517 20254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:07.395882 20254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:07.414958 20254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34093
I20260619 13:12:07.414983 20319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34093 every 8 connection(s)
I20260619 13:12:07.416344 20254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:07.418063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20254
I20260619 13:12:07.418313 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:07.418912 388 external_mini_cluster.cc:1428] Setting key aca26e1d074d1f94848caa6cbd4b6f43
I20260619 13:12:07.421849 20320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:07.432862 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:07.421003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.97.62:34093)
0619 13:12:07.422210 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:07.422222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:07.422264 (+ 42us) server_negotiation.cc:407] Connection header received
0619 13:12:07.423921 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:07.423949 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:07.424422 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:07.424851 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:07.425450 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:07.426851 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:07.428255 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:07.428942 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:07.429149 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:07.430509 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:07.430517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:07.430531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:07.431130 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:07.431211 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:07.431232 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:07.431376 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:07.431593 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 13:12:07.431913 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":400,"threads_started":1}
I20260619 13:12:07.434054 20320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: Bootstrap starting.
I20260619 13:12:07.437801 20320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:07.438956 20320 log.cc:824] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:07.442286 20320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: No bootstrap required, opened a new log
I20260619 13:12:07.450469 20320 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260619 13:12:07.450894 20320 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:07.450984 20320 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf8bbeea57544c3861a7bdb1811925e, State: Initialized, Role: FOLLOWER
I20260619 13:12:07.451715 20320 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260619 13:12:07.452067 20320 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:07.452246 20320 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:07.452471 20320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:07.454968 20320 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260619 13:12:07.455520 20320 leader_election.cc:302] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf8bbeea57544c3861a7bdb1811925e; no voters:
I20260619 13:12:07.456122 20320 leader_election.cc:288] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:07.456311 20325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:07.457098 20325 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [term 1 LEADER]: Becoming Leader. State: Replica: acf8bbeea57544c3861a7bdb1811925e, State: Running, Role: LEADER
I20260619 13:12:07.457729 20325 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260619 13:12:07.458600 20320 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:07.460841 20326 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf8bbeea57544c3861a7bdb1811925e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } } }
I20260619 13:12:07.461370 20326 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:07.461736 20327 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf8bbeea57544c3861a7bdb1811925e. Latest consensus state: current_term: 1 leader_uuid: "acf8bbeea57544c3861a7bdb1811925e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8bbeea57544c3861a7bdb1811925e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } } }
I20260619 13:12:07.462118 20327 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:07.467420 20331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:07.473239 20331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:07.482523 20331 catalog_manager.cc:1347] Generated new cluster ID: 9d7f80b7fc2749a0b64314c276d66b63
I20260619 13:12:07.482651 20331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:07.513921 20331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:07.515862 20331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:07.527076 20331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: Generated new TSK 0
I20260619 13:12:07.528239 20331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:07.546023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:12:07.887252 20344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:07.887585 20344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:07.887753 20344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:07.895174 20344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:07.895329 20344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:07.895515 20344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:12:07.905136 20344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:07.906543 20344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:07.908342 20344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:07.921393 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:07.921499 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:07.921756 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:07.922595 20344 server_base.cc:1034] running on GCE node
I20260619 13:12:07.923218 20344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:07.924185 20344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:07.925426 20344 hybrid_clock.cc:648] HybridClock initialized: now 1781874727925365 us; error 58 us; skew 500 ppm
I20260619 13:12:07.925817 20344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:07.928236 20344 webserver.cc:460] Webserver started at http://127.0.97.1:35739/ using document root <none> and password file <none>
I20260619 13:12:07.928961 20344 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:07.929097 20344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:07.929428 20344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:07.932461 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data/instance:
uuid: "4da9d4b2cef448e1b053535781bb1db3"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "1f20d099744e25adf4ea200adb69a2df"
server_key_iv: "d42201f50a043ce1381d74d7fd831e89"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.933260 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4da9d4b2cef448e1b053535781bb1db3"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "1f20d099744e25adf4ea200adb69a2df"
server_key_iv: "d42201f50a043ce1381d74d7fd831e89"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.938503 20344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:12:07.942183 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:07.943503 20344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:07.943672 20344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal
uuid: "4da9d4b2cef448e1b053535781bb1db3"
format_stamp: "Formatted at 2026-06-19 13:12:07 on dist-test-slave-q11v"
server_key: "1f20d099744e25adf4ea200adb69a2df"
server_key_iv: "d42201f50a043ce1381d74d7fd831e89"
server_key_version: "encryptionkey@0"
I20260619 13:12:07.943905 20344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:07.956267 20344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:07.957111 20344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:07.957401 20344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:07.958397 20344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:07.960109 20344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:07.960234 20344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:07.960356 20344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:07.960441 20344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:07.993970 20344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39883
I20260619 13:12:07.994064 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39883 every 8 connection(s)
I20260619 13:12:07.995334 20344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:12:08.004487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20344
I20260619 13:12:08.004676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:12:08.005174 388 external_mini_cluster.cc:1428] Setting key 350afab35e640f87dec00a20f14388f5
I20260619 13:12:08.007966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:12:08.008239 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:07.997892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53417 (local address 127.0.97.62:34093)
0619 13:12:07.998351 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:07.998357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:07.999675 (+ 1318us) server_negotiation.cc:407] Connection header received
0619 13:12:08.001404 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.001408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.001481 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.001576 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:08.002687 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.003401 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.005093 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.005714 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.005845 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.007385 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:08.007389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:08.007392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:08.007409 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:08.007447 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:08.007459 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:08.007557 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.007791 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.007892 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260619 13:12:08.009274 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:07.998587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34093 (local address 127.0.97.1:53417)
0619 13:12:07.999528 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:07.999566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.001161 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.001835 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.001843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.001939 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:08.002531 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.002549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.003556 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.003559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.004946 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.004958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.005977 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.005988 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.006242 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.007057 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:08.007060 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:08.007105 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:08.007185 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:08.007554 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:08.007567 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:08.007578 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 13:12:08.007771 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 13:12:08.008187 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":305,"threads_started":1}
I20260619 13:12:08.010905 20471 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34093
I20260619 13:12:08.011219 20471 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:08.011974 20471 heartbeater.cc:511] Master 127.0.97.62:34093 requested a full tablet report, sending...
I20260619 13:12:08.014278 20285 ts_manager.cc:194] Registered new tserver with Master: 4da9d4b2cef448e1b053535781bb1db3 (127.0.97.1:39883)
I20260619 13:12:08.016284 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53417
W20260619 13:12:08.338369 20475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:08.338652 20475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:08.338804 20475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:08.346380 20475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:08.346482 20475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:08.346685 20475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:12:08.355921 20475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:08.357206 20475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:08.359054 20475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:08.372596 20480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:08.372624 20483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:08.372655 20481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:08.373256 20475 server_base.cc:1034] running on GCE node
I20260619 13:12:08.373966 20475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:08.375052 20475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:08.376353 20475 hybrid_clock.cc:648] HybridClock initialized: now 1781874728376244 us; error 72 us; skew 500 ppm
I20260619 13:12:08.376668 20475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:08.378940 20475 webserver.cc:460] Webserver started at http://127.0.97.2:40469/ using document root <none> and password file <none>
I20260619 13:12:08.379654 20475 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:08.379786 20475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:08.380101 20475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:08.382982 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data/instance:
uuid: "bea49689c67041f28bb506fcd5fe3a50"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "1bdcb3d6fdbf01fbc6c1abc1890b7f7d"
server_key_iv: "716b4ba1865347ca356feb4b02d4db4a"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.383822 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bea49689c67041f28bb506fcd5fe3a50"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "1bdcb3d6fdbf01fbc6c1abc1890b7f7d"
server_key_iv: "716b4ba1865347ca356feb4b02d4db4a"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.388998 20475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:12:08.393064 20489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.394434 20475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:08.394613 20475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal
uuid: "bea49689c67041f28bb506fcd5fe3a50"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "1bdcb3d6fdbf01fbc6c1abc1890b7f7d"
server_key_iv: "716b4ba1865347ca356feb4b02d4db4a"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.394846 20475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:08.407506 20475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:08.408303 20475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:08.408589 20475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:08.409509 20475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:08.410976 20475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:08.411106 20475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.411232 20475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:08.411317 20475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.448308 20475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38665
I20260619 13:12:08.448357 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38665 every 8 connection(s)
I20260619 13:12:08.449635 20475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:12:08.454927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20475
I20260619 13:12:08.455098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:12:08.455559 388 external_mini_cluster.cc:1428] Setting key 31f699fcd7952bd1eceb81eba3215557
I20260619 13:12:08.458604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:12:08.463011 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.452172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51285 (local address 127.0.97.62:34093)
0619 13:12:08.452347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.452352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.453774 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 13:12:08.455649 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.455655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.455743 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.455856 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:08.457130 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.458144 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.459907 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.460648 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.460795 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.462102 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:08.462110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:08.462118 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:08.462143 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:08.462188 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:08.462206 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:08.462300 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.462501 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.462615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:12:08.463907 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.452679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34093 (local address 127.0.97.2:51285)
0619 13:12:08.453556 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:08.453641 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.455364 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.456065 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.456073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.456206 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:08.456967 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.456992 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.458310 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.458318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.459763 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.459775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.460843 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.460848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.461043 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.461767 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:08.461770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:08.461823 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:08.461903 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:08.462302 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:08.462309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:08.462315 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:12:08.462460 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 13:12:08.462766 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":276,"threads_started":1}
I20260619 13:12:08.465430 20602 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34093
I20260619 13:12:08.465793 20602 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:08.466706 20602 heartbeater.cc:511] Master 127.0.97.62:34093 requested a full tablet report, sending...
I20260619 13:12:08.468238 20285 ts_manager.cc:194] Registered new tserver with Master: bea49689c67041f28bb506fcd5fe3a50 (127.0.97.2:38665)
I20260619 13:12:08.469511 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51285
W20260619 13:12:08.785378 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:08.785674 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:08.785760 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:08.793018 20605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:08.793120 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:08.793228 20605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:12:08.802451 20605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:08.803807 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:08.805554 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:08.818185 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:08.818135 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:08.818238 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:08.819307 20605 server_base.cc:1034] running on GCE node
I20260619 13:12:08.819968 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:08.820937 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:08.822155 20605 hybrid_clock.cc:648] HybridClock initialized: now 1781874728822092 us; error 60 us; skew 500 ppm
I20260619 13:12:08.822556 20605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:12:08.824817 20605 webserver.cc:460] Webserver started at http://127.0.97.3:33857/ using document root <none> and password file <none>
I20260619 13:12:08.825515 20605 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:08.825655 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:08.826023 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:08.828861 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data/instance:
uuid: "1960bfe3d16241bb99710cb87710d340"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "85a8ba12eff229ffd3224835c69834af"
server_key_iv: "269f3a791a821470bf15f28d7558bb4e"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.829710 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1960bfe3d16241bb99710cb87710d340"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "85a8ba12eff229ffd3224835c69834af"
server_key_iv: "269f3a791a821470bf15f28d7558bb4e"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.834712 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:12:08.838040 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.839389 20605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:12:08.839568 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal
uuid: "1960bfe3d16241bb99710cb87710d340"
format_stamp: "Formatted at 2026-06-19 13:12:08 on dist-test-slave-q11v"
server_key: "85a8ba12eff229ffd3224835c69834af"
server_key_iv: "269f3a791a821470bf15f28d7558bb4e"
server_key_version: "encryptionkey@0"
I20260619 13:12:08.839785 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:08.876029 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:08.876931 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:08.877221 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:08.878404 20605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:12:08.879935 20605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:12:08.880057 20605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.880200 20605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:12:08.880290 20605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:08.913782 20605 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43381
I20260619 13:12:08.913842 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43381 every 8 connection(s)
I20260619 13:12:08.915132 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:12:08.925215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20605
I20260619 13:12:08.925400 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:12:08.925900 388 external_mini_cluster.cc:1428] Setting key af829038c5d803d5f908621fecb21e85
I20260619 13:12:08.932376 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.918133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34509 (local address 127.0.97.62:34093)
0619 13:12:08.918324 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.918329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.919764 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 13:12:08.921627 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.921634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.921732 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.921847 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:08.922942 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.923687 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.925252 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.925852 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.925998 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.931552 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:08.931557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:08.931560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:08.931579 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:08.931612 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:08.931624 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:08.931722 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.931951 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.932055 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:12:08.933480 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.918668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34093 (local address 127.0.97.3:34509)
0619 13:12:08.919586 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:08.919632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.921308 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.922092 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.922101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.922191 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:08.922760 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.922777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.923834 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.923846 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.925090 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.925102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.926093 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.926100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.926385 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.931203 (+ 4818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:12:08.931207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:08.931251 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:12:08.931342 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:08.931722 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:08.931736 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:08.931746 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:12:08.931936 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 13:12:08.932368 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260619 13:12:08.934906 20732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34093
I20260619 13:12:08.935189 20732 heartbeater.cc:464] Registering TS with master...
I20260619 13:12:08.935920 20732 heartbeater.cc:511] Master 127.0.97.62:34093 requested a full tablet report, sending...
I20260619 13:12:08.937290 20285 ts_manager.cc:194] Registered new tserver with Master: 1960bfe3d16241bb99710cb87710d340 (127.0.97.3:43381)
I20260619 13:12:08.938172 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34509
I20260619 13:12:08.940821 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:12:08.950675 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.944980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.0.97.62:34093)
0619 13:12:08.945186 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.945194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.945557 (+ 363us) server_negotiation.cc:407] Connection header received
0619 13:12:08.945805 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.945816 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.945889 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.945999 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:08.946475 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.947160 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.948624 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.949034 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.949129 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.950144 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:08.950149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:08.950152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:08.950169 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:08.950198 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:08.950210 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:08.950280 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.950390 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.950466 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:12:08.954337 20285 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:12:08.961290 20285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:12:08.991276 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.973876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.97.3:43381)
0619 13:12:08.977982 (+ 4106us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.977995 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.978041 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:12:08.978167 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.978179 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.978514 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.979734 (+ 1220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:08.981341 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.983509 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.988516 (+ 5007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.990034 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.990179 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.990479 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.990725 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.990897 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3875,"thread_start_us":706,"threads_started":1}
I20260619 13:12:08.991246 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.973542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43381 (local address 127.0.0.1:38238)
0619 13:12:08.975103 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:08.975147 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.975309 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.978821 (+ 3512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.978833 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.978877 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:08.979408 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.979434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.983724 (+ 4290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.983733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.986956 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.986979 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.990248 (+ 3269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.990255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.990471 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.990525 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 13:12:08.990687 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 13:12:08.990900 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":15211,"threads_started":1}
I20260619 13:12:08.995479 20667 tablet_service.cc:1467] Processing CreateTablet for tablet dff755dea4d24520acfefa6fa826c8bb (DEFAULT_TABLE table=test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:08.996225 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.974411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39883 (local address 127.0.0.1:42086)
0619 13:12:08.976050 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:08.976068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.977363 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.989718 (+ 12355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.989722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.989751 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:08.990102 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.990116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.992456 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.992464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.994155 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.994167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.995511 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.995517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:08.995676 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.995707 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:12:08.995791 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:12:08.995921 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"thread_start_us":376,"threads_started":1}
I20260619 13:12:08.996459 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.974728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.97.1:39883)
0619 13:12:08.989060 (+ 14332us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.989069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.989097 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:12:08.989198 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.989207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.989470 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.989718 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:08.990319 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.992321 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.994348 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:08.995315 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:08.995505 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:08.995804 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:08.995875 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 13:12:08.996069 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14167,"thread_start_us":14055,"threads_started":1}
I20260619 13:12:08.997119 20667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff755dea4d24520acfefa6fa826c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:08.999677 20406 tablet_service.cc:1467] Processing CreateTablet for tablet dff755dea4d24520acfefa6fa826c8bb (DEFAULT_TABLE table=test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:09.001066 20406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff755dea4d24520acfefa6fa826c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:09.006551 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.975559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38665 (local address 127.0.0.1:40012)
0619 13:12:08.988455 (+ 12896us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:08.988481 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:08.988642 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:08.990534 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:08.990541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:08.990569 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:08.990978 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:08.990990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.002217 (+ 11227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.002222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.003827 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:09.003842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.005733 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.005740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.005926 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.005961 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:12:09.006053 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 13:12:09.006188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12727,"mutex_wait_us":1,"thread_start_us":655,"threads_started":1}
I20260619 13:12:09.006829 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:08.976050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.0.97.2:38665)
0619 13:12:08.988396 (+ 12346us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:08.988408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:08.988635 (+ 227us) server_negotiation.cc:407] Connection header received
0619 13:12:08.989773 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:08.989784 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:08.990074 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:08.990265 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:08.991175 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.001987 (+ 10812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.004030 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.005493 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.005690 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.005937 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.006107 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.006276 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12156,"thread_start_us":12352,"threads_started":1}
I20260619 13:12:09.010596 20537 tablet_service.cc:1467] Processing CreateTablet for tablet dff755dea4d24520acfefa6fa826c8bb (DEFAULT_TABLE table=test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:12:09.012019 20537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff755dea4d24520acfefa6fa826c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:09.013827 20751 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Bootstrap starting.
I20260619 13:12:09.016929 20751 tablet_bootstrap.cc:654] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:09.018251 20751 log.cc:824] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:09.018767 20750 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Bootstrap starting.
I20260619 13:12:09.020486 20471 heartbeater.cc:503] Master 127.0.97.62:34093 was elected leader, sending a full tablet report...
I20260619 13:12:09.022224 20751 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: No bootstrap required, opened a new log
I20260619 13:12:09.022521 20751 ts_tablet_manager.cc:1397] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260619 13:12:09.026206 20750 tablet_bootstrap.cc:654] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:09.027559 20750 log.cc:824] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:09.028976 20754 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Bootstrap starting.
I20260619 13:12:09.030738 20750 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: No bootstrap required, opened a new log
I20260619 13:12:09.031028 20750 ts_tablet_manager.cc:1397] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260619 13:12:09.031950 20754 tablet_bootstrap.cc:654] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:09.032577 20751 raft_consensus.cc:348] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.033027 20751 raft_consensus.cc:374] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:09.033020 20754 log.cc:824] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:09.033140 20751 raft_consensus.cc:729] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da9d4b2cef448e1b053535781bb1db3, State: Initialized, Role: FOLLOWER
I20260619 13:12:09.033833 20751 consensus_queue.cc:260] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.035626 20751 ts_tablet_manager.cc:1428] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260619 13:12:09.038882 20754 tablet_bootstrap.cc:492] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: No bootstrap required, opened a new log
I20260619 13:12:09.039182 20754 ts_tablet_manager.cc:1397] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 13:12:09.042066 20750 raft_consensus.cc:348] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.042551 20750 raft_consensus.cc:374] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:09.042656 20750 raft_consensus.cc:729] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960bfe3d16241bb99710cb87710d340, State: Initialized, Role: FOLLOWER
I20260619 13:12:09.043278 20750 consensus_queue.cc:260] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.044451 20732 heartbeater.cc:503] Master 127.0.97.62:34093 was elected leader, sending a full tablet report...
I20260619 13:12:09.044982 20750 ts_tablet_manager.cc:1428] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 13:12:09.048246 20754 raft_consensus.cc:348] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.048642 20754 raft_consensus.cc:374] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:09.048753 20754 raft_consensus.cc:729] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea49689c67041f28bb506fcd5fe3a50, State: Initialized, Role: FOLLOWER
I20260619 13:12:09.049314 20754 consensus_queue.cc:260] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.050437 20602 heartbeater.cc:503] Master 127.0.97.62:34093 was elected leader, sending a full tablet report...
I20260619 13:12:09.050981 20754 ts_tablet_manager.cc:1428] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 13:12:09.113300 20757 raft_consensus.cc:482] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:12:09.113673 20757 raft_consensus.cc:504] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.115453 20757 leader_election.cc:288] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da9d4b2cef448e1b053535781bb1db3 (127.0.97.1:39883), bea49689c67041f28bb506fcd5fe3a50 (127.0.97.2:38665)
I20260619 13:12:09.124029 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.116503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42655 (local address 127.0.97.2:38665)
0619 13:12:09.117062 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.117070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.117088 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:12:09.117157 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.117162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.117275 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.117371 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:09.118303 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.119407 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.122340 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.123087 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.123237 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.123330 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.123631 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.123758 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417}
I20260619 13:12:09.124042 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.116834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40757 (local address 127.0.97.1:39883)
0619 13:12:09.117077 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.117086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.117107 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:12:09.117192 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.117198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.117312 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.117441 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:12:09.118317 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.119548 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.122312 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.123070 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.123237 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.123390 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.123639 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.123784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:12:09.124094 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.115760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39883 (local address 127.0.97.3:40757)
0619 13:12:09.115940 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:09.115957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:09.116057 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:09.117666 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:09.117674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:09.117745 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:09.118097 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:09.118116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.119662 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.119666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.122130 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:09.122145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.123244 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.123250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.123471 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.123495 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:12:09.123611 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 13:12:09.123754 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":186,"spinlock_wait_cycles":7552}
I20260619 13:12:09.124140 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.116209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38665 (local address 127.0.97.3:42655)
0619 13:12:09.116876 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:09.116898 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:09.117024 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:09.117540 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:09.117545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:09.117572 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:12:09.118075 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:09.118088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.119595 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.119600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.122130 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:09.122145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.123259 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.123266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:09.123465 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.123490 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:12:09.123592 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:12:09.123725 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":127,"thread_start_us":314,"threads_started":1}
I20260619 13:12:09.125206 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff755dea4d24520acfefa6fa826c8bb" candidate_uuid: "1960bfe3d16241bb99710cb87710d340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da9d4b2cef448e1b053535781bb1db3" is_pre_election: true
I20260619 13:12:09.125435 20557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff755dea4d24520acfefa6fa826c8bb" candidate_uuid: "1960bfe3d16241bb99710cb87710d340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea49689c67041f28bb506fcd5fe3a50" is_pre_election: true
I20260619 13:12:09.125777 20426 raft_consensus.cc:2454] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1960bfe3d16241bb99710cb87710d340 in term 0.
I20260619 13:12:09.126143 20557 raft_consensus.cc:2454] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1960bfe3d16241bb99710cb87710d340 in term 0.
I20260619 13:12:09.126451 20620 leader_election.cc:302] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960bfe3d16241bb99710cb87710d340, 4da9d4b2cef448e1b053535781bb1db3; no voters:
I20260619 13:12:09.126860 20757 raft_consensus.cc:2781] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:12:09.127022 20757 raft_consensus.cc:482] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:12:09.127140 20757 raft_consensus.cc:3037] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:09.129390 20757 raft_consensus.cc:504] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.130056 20757 leader_election.cc:288] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [CANDIDATE]: Term 1 election: Requested vote from peers 4da9d4b2cef448e1b053535781bb1db3 (127.0.97.1:39883), bea49689c67041f28bb506fcd5fe3a50 (127.0.97.2:38665)
I20260619 13:12:09.130571 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff755dea4d24520acfefa6fa826c8bb" candidate_uuid: "1960bfe3d16241bb99710cb87710d340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da9d4b2cef448e1b053535781bb1db3"
I20260619 13:12:09.130648 20557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff755dea4d24520acfefa6fa826c8bb" candidate_uuid: "1960bfe3d16241bb99710cb87710d340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea49689c67041f28bb506fcd5fe3a50"
I20260619 13:12:09.130826 20426 raft_consensus.cc:3037] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:09.130869 20557 raft_consensus.cc:3037] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:09.133097 20557 raft_consensus.cc:2454] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1960bfe3d16241bb99710cb87710d340 in term 1.
I20260619 13:12:09.133503 20620 leader_election.cc:302] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960bfe3d16241bb99710cb87710d340, bea49689c67041f28bb506fcd5fe3a50; no voters:
I20260619 13:12:09.133494 20426 raft_consensus.cc:2454] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1960bfe3d16241bb99710cb87710d340 in term 1.
I20260619 13:12:09.133795 20757 raft_consensus.cc:2781] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:09.134292 20757 raft_consensus.cc:686] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 1 LEADER]: Becoming Leader. State: Replica: 1960bfe3d16241bb99710cb87710d340, State: Running, Role: LEADER
I20260619 13:12:09.134769 20757 consensus_queue.cc:237] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } }
I20260619 13:12:09.139339 20284 catalog_manager.cc:5495] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1960bfe3d16241bb99710cb87710d340 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1960bfe3d16241bb99710cb87710d340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960bfe3d16241bb99710cb87710d340" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20260619 13:12:09.154795 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.149589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.97.3:43381)
0619 13:12:09.149855 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.149863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.149887 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:12:09.150127 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.150131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.150220 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.150388 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:09.150711 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.151450 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.152841 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.153234 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.153333 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.153763 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:09.154232 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:09.154361 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.154463 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.154523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:12:09.162750 20426 raft_consensus.cc:1261] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 1 FOLLOWER]: Refusing update from remote peer 1960bfe3d16241bb99710cb87710d340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:09.162750 20557 raft_consensus.cc:1261] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 1 FOLLOWER]: Refusing update from remote peer 1960bfe3d16241bb99710cb87710d340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:09.163761 20760 consensus_queue.cc:1035] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da9d4b2cef448e1b053535781bb1db3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:09.164143 20757 consensus_queue.cc:1035] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea49689c67041f28bb506fcd5fe3a50" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:12:09.167151 20733 tablet.cc:2369] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:09.190440 20284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:09.190752 20284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:12:09.194658 20284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acf8bbeea57544c3861a7bdb1811925e: Sending DeleteTablet for 3 replicas of tablet dff755dea4d24520acfefa6fa826c8bb
I20260619 13:12:09.195940 20667 tablet_service.cc:1514] Processing DeleteTablet for tablet dff755dea4d24520acfefa6fa826c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:09 UTC) from {username='slave'} at 127.0.0.1:38238
I20260619 13:12:09.196082 20406 tablet_service.cc:1514] Processing DeleteTablet for tablet dff755dea4d24520acfefa6fa826c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:09 UTC) from {username='slave'} at 127.0.0.1:42086
I20260619 13:12:09.196290 20537 tablet_service.cc:1514] Processing DeleteTablet for tablet dff755dea4d24520acfefa6fa826c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:12:09 UTC) from {username='slave'} at 127.0.0.1:40012
I20260619 13:12:09.197075 20774 tablet_replica.cc:330] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: stopping tablet replica
I20260619 13:12:09.197525 20773 tablet_replica.cc:330] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: stopping tablet replica
I20260619 13:12:09.197695 20774 raft_consensus.cc:2229] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:09.197808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20344
I20260619 13:12:09.198048 20773 raft_consensus.cc:2229] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:12:09.198472 20774 raft_consensus.cc:2258] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:09.198910 20773 raft_consensus.cc:2258] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:09.201505 20773 ts_tablet_manager.cc:1905] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:09.204051 20774 ts_tablet_manager.cc:1905] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:09.210878 20773 ts_tablet_manager.cc:1918] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:09.211076 20773 log.cc:1196] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-2/wal/wals/dff755dea4d24520acfefa6fa826c8bb
I20260619 13:12:09.211669 20773 ts_tablet_manager.cc:1939] T dff755dea4d24520acfefa6fa826c8bb P 1960bfe3d16241bb99710cb87710d340: Deleting consensus metadata
I20260619 13:12:09.212311 20774 ts_tablet_manager.cc:1918] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:09.212570 20774 log.cc:1196] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-0/wal/wals/dff755dea4d24520acfefa6fa826c8bb
I20260619 13:12:09.213213 20774 ts_tablet_manager.cc:1939] T dff755dea4d24520acfefa6fa826c8bb P 4da9d4b2cef448e1b053535781bb1db3: Deleting consensus metadata
I20260619 13:12:09.213773 20272 catalog_manager.cc:4841] TS 1960bfe3d16241bb99710cb87710d340 (127.0.97.3:43381): tablet dff755dea4d24520acfefa6fa826c8bb (table test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]) successfully deleted
I20260619 13:12:09.215611 20269 catalog_manager.cc:4841] TS 4da9d4b2cef448e1b053535781bb1db3 (127.0.97.1:39883): tablet dff755dea4d24520acfefa6fa826c8bb (table test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]) successfully deleted
I20260619 13:12:09.215996 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.199650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.0.97.1:39883)
0619 13:12:09.200531 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.200537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.207310 (+ 6773us) server_negotiation.cc:407] Connection header received
0619 13:12:09.207539 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.207547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.207655 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.207763 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:09.208429 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.209669 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.211645 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.212470 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.212584 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.214236 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:09.214248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:09.214278 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:09.215077 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:09.215137 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:09.215161 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:09.215268 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.215529 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.215665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758}
I20260619 13:12:09.216261 20775 tablet_replica.cc:330] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: stopping tablet replica
I20260619 13:12:09.216742 20775 raft_consensus.cc:2229] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:12:09.217010 20466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:09.217296 20775 raft_consensus.cc:2258] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:12:09.223552 20775 ts_tablet_manager.cc:1905] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:12:09.230898 20775 ts_tablet_manager.cc:1918] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:12:09.231101 20775 log.cc:1196] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Deleting WAL directory at /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874405339864-388-0/minicluster-data/ts-1/wal/wals/dff755dea4d24520acfefa6fa826c8bb
I20260619 13:12:09.231659 20775 ts_tablet_manager.cc:1939] T dff755dea4d24520acfefa6fa826c8bb P bea49689c67041f28bb506fcd5fe3a50: Deleting consensus metadata
I20260619 13:12:09.233705 20269 catalog_manager.cc:4841] TS bea49689c67041f28bb506fcd5fe3a50 (127.0.97.2:38665): tablet dff755dea4d24520acfefa6fa826c8bb (table test-table [id=d5acb7f6ecd846ff86ba48db1dc5acb6]) successfully deleted
I20260619 13:12:09.310576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20344
I20260619 13:12:09.329097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20475
I20260619 13:12:09.339571 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.329979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.0.97.2:38665)
0619 13:12:09.330146 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.330151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.330168 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:12:09.330369 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.330373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.330489 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.330574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:09.331024 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.331755 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.333704 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.334204 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.334301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.338195 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:09.338204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:09.338219 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:09.338803 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:09.338857 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:09.338878 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:09.339017 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.339148 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.339262 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:12:09.340224 20597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:09.457828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20475
I20260619 13:12:09.475699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20605
I20260619 13:12:09.483871 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:09.476667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.0.97.3:43381)
0619 13:12:09.476856 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:09.476860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:09.476879 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:12:09.476991 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:09.476995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:09.477069 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:09.477163 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:09.477692 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.478486 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.480409 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:09.480867 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:09.480960 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:09.482278 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:09.482287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:09.482305 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:09.482977 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:09.483028 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:09.483048 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:09.483133 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:09.483377 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 13:12:09.483463 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:12:09.484639 20727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:09.607821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20605
I20260619 13:12:09.627530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20254
I20260619 13:12:09.628599 20315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:09.737968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20254
2026-06-19T13:12:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2839 ms)
[----------] 6 tests from EncryptionPolicyTest (21817 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:09 dist-test-slave-q11v krb5kdc[20783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:09 dist-test-slave-q11v krb5kdc[20783](info): set up 2 sockets
Jun 19 13:12:09 dist-test-slave-q11v krb5kdc[20783](info): commencing operation
krb5kdc: starting...
W20260619 13:12:10.854095 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:11.112778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:11.443665 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:11.443974 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:11.444139 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:11.451542 20796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:11.451663 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:11.451766 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:11.451880 20796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:11.451961 20796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:11.462265 20796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:11.463672 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:11.465008 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:11.477180 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:11.477210 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:11.477133 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:11.477981 20796 server_base.cc:1034] running on GCE node
I20260619 13:12:11.478834 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:11.480329 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:11.481575 20796 hybrid_clock.cc:648] HybridClock initialized: now 1781874731481534 us; error 34 us; skew 500 ppm
I20260619 13:12:11.482002 20796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:11.485131 20796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:11.486727 20796 webserver.cc:460] Webserver started at http://127.0.0.1:33037/ using document root <none> and password file <none>
I20260619 13:12:11.487437 20796 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:11.487584 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:11.487900 20796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:11.490970 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "c03e72b90a6b4a5f963ca3c4d7057238"
format_stamp: "Formatted at 2026-06-19 13:12:11 on dist-test-slave-q11v"
server_key: "b24be44257effa33681510e8a27265c1"
server_key_iv: "7e7795d7ebd41ca4b8069121aa0785db"
server_key_version: "encryptionkey@0"
I20260619 13:12:11.491757 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "c03e72b90a6b4a5f963ca3c4d7057238"
format_stamp: "Formatted at 2026-06-19 13:12:11 on dist-test-slave-q11v"
server_key: "b24be44257effa33681510e8a27265c1"
server_key_iv: "7e7795d7ebd41ca4b8069121aa0785db"
server_key_version: "encryptionkey@0"
I20260619 13:12:11.497299 20796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:12:11.500973 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:11.502444 20796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:11.502651 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "c03e72b90a6b4a5f963ca3c4d7057238"
format_stamp: "Formatted at 2026-06-19 13:12:11 on dist-test-slave-q11v"
server_key: "b24be44257effa33681510e8a27265c1"
server_key_iv: "7e7795d7ebd41ca4b8069121aa0785db"
server_key_version: "encryptionkey@0"
I20260619 13:12:11.502873 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:11.517479 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:11.521796 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:11.522086 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:11.540969 20796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43413
I20260619 13:12:11.541005 20862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43413 every 8 connection(s)
I20260619 13:12:11.542479 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:11.546254 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:11.547566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20796
I20260619 13:12:11.547784 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:11.548566 388 external_mini_cluster.cc:1428] Setting key 9861ce687dc5d019423f3ac288584feb
I20260619 13:12:11.559230 20863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238: Bootstrap starting.
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:12:11.562736 20863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:11.563987 20863 log.cc:824] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:11.567091 20863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238: No bootstrap required, opened a new log
I20260619 13:12:11.571547 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:11.550600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35367 (local address 127.0.0.1:43413)
0619 13:12:11.551350 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:11.551362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:11.551406 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:12:11.552997 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:11.553019 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:11.553422 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:11.553890 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:11.555302 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:11.556638 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:11.558060 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:11.558756 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:11.558973 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:11.563580 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:11.563588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:11.563604 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:11.563641 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:11.567190 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:11.568098 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:11.568104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:11.568110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:11.568283 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:11.568773 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:11.568777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:11.568779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:11.569305 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:11.569701 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:11.570201 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 13:12:11.570539 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":332,"threads_started":1}
I20260619 13:12:11.575132 20863 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260619 13:12:11.575534 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:11.575573 20863 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:11.575673 20863 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03e72b90a6b4a5f963ca3c4d7057238, State: Initialized, Role: FOLLOWER
I20260619 13:12:11.576222 20863 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260619 13:12:11.576457 20863 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:11.576587 20863 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:11.576761 20863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:11.579128 20863 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260619 13:12:11.579670 20863 leader_election.cc:302] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03e72b90a6b4a5f963ca3c4d7057238; no voters:
I20260619 13:12:11.580251 20863 leader_election.cc:288] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:11.580380 20872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:11.581089 20872 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [term 1 LEADER]: Becoming Leader. State: Replica: c03e72b90a6b4a5f963ca3c4d7057238, State: Running, Role: LEADER
I20260619 13:12:11.581691 20872 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260619 13:12:11.584211 20874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } } }
I20260619 13:12:11.584581 20874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:11.588323 20873 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03e72b90a6b4a5f963ca3c4d7057238. Latest consensus state: current_term: 1 leader_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03e72b90a6b4a5f963ca3c4d7057238" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } } }
I20260619 13:12:11.588651 20873 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:11.590031 20863 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:11.596505 20877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:11.602589 20877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:11.605996 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:11.580625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38397 (local address 127.0.0.1:43413)
0619 13:12:11.580834 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:11.580842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:11.584544 (+ 3702us) server_negotiation.cc:407] Connection header received
0619 13:12:11.585073 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:11.585083 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:11.585185 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:11.585271 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:11.588854 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:11.590014 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:11.592167 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:11.593107 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:11.593234 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:11.597359 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:11.597371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:11.597375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:11.597422 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:11.601458 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:11.602611 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:11.602618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:11.602622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:11.602855 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:11.603497 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:11.603503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:11.603507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:11.603951 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:11.605184 (+ 1233us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:11.605377 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 13:12:11.605584 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:12:11.614050 20877 catalog_manager.cc:1347] Generated new cluster ID: 84afb0a465f24cf6b9ceb04c89924846
I20260619 13:12:11.614182 20877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:11.627149 20877 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:11.628490 20877 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:11.634104 20877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03e72b90a6b4a5f963ca3c4d7057238: Generated new TSK 0
I20260619 13:12:11.635115 20877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:11.648031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20796
I20260619 13:12:11.649180 20858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:11.741369 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20796
W20260619 13:12:11.758944 20781 connection.cc:535] client connection to 127.0.0.1:43413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43413): Connection reset by peer (error 104)
2026-06-19T13:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20896](info): set up 2 sockets
Jun 19 13:12:11 dist-test-slave-q11v krb5kdc[20896](info): commencing operation
krb5kdc: starting...
W20260619 13:12:12.853888 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:13 dist-test-slave-q11v krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:13.106989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:13.440200 20909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:13.440609 20909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:13.440775 20909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:13.448359 20909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:13.448529 20909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:13.448637 20909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:13.448746 20909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:13.448829 20909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:13.458926 20909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38147
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:13.460392 20909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:13.461796 20909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:13.474032 20914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:13.474093 20917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:13.474071 20915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:13.474880 20909 server_base.cc:1034] running on GCE node
I20260619 13:12:13.475696 20909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:13.477212 20909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:13.478446 20909 hybrid_clock.cc:648] HybridClock initialized: now 1781874733478386 us; error 53 us; skew 500 ppm
I20260619 13:12:13.478816 20909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:13 dist-test-slave-q11v krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:13.482050 20909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:13.483517 20909 webserver.cc:460] Webserver started at http://127.0.0.1:39351/ using document root <none> and password file <none>
I20260619 13:12:13.484220 20909 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:13.484369 20909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:13.484678 20909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:13.487766 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "1257af6aeb1e4002b1d0a0233241f2a4"
format_stamp: "Formatted at 2026-06-19 13:12:13 on dist-test-slave-q11v"
server_key: "c41be1e91aa16b683e70d6345d09d004"
server_key_iv: "e0afc10269f31caed58e32c26ae932e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:13.488528 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "1257af6aeb1e4002b1d0a0233241f2a4"
format_stamp: "Formatted at 2026-06-19 13:12:13 on dist-test-slave-q11v"
server_key: "c41be1e91aa16b683e70d6345d09d004"
server_key_iv: "e0afc10269f31caed58e32c26ae932e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:13.493722 20909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 13:12:13.497103 20924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:13.498497 20909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:12:13.498682 20909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "1257af6aeb1e4002b1d0a0233241f2a4"
format_stamp: "Formatted at 2026-06-19 13:12:13 on dist-test-slave-q11v"
server_key: "c41be1e91aa16b683e70d6345d09d004"
server_key_iv: "e0afc10269f31caed58e32c26ae932e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:13.498907 20909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:13.518630 20909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:13.523276 20909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:13.523589 20909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:13.542160 20909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38147
I20260619 13:12:13.542173 20975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38147 every 8 connection(s)
I20260619 13:12:13.543570 20909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:13.547427 20976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:13.552193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 20909
I20260619 13:12:13.552377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:13.552814 388 external_mini_cluster.cc:1428] Setting key ee31cbc3308b4142145afc1e7723fa2e
I20260619 13:12:13.560438 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: Bootstrap starting.
I20260619 13:12:13.564172 20976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:13.565464 20976 log.cc:824] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:13.568775 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:13.554691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54989 (local address 127.0.0.1:38147)
0619 13:12:13.555490 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:13.555507 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:13.555564 (+ 57us) server_negotiation.cc:407] Connection header received
0619 13:12:13.558634 (+ 3070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:13.558663 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:13.559189 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:13.559641 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:13.560169 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:13.562153 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:13.563563 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:13.564656 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:13.564829 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:13.566068 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:13.566084 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:13.566109 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:13.566763 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:13.566871 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:13.566907 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:13.567063 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:13.567320 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 13:12:13.567500 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":345,"threads_started":1}
I20260619 13:12:13.568842 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: No bootstrap required, opened a new log
I20260619 13:12:13.572876 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:13.577744 20976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } }
I20260619 13:12:13.578142 20976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:13.578293 20976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1257af6aeb1e4002b1d0a0233241f2a4, State: Initialized, Role: FOLLOWER
I20260619 13:12:13.578949 20976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } }
I20260619 13:12:13.579218 20976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:13.579370 20976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:13.579566 20976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:13.581874 20976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } }
I20260619 13:12:13.582393 20976 leader_election.cc:302] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1257af6aeb1e4002b1d0a0233241f2a4; no voters:
I20260619 13:12:13.583071 20976 leader_election.cc:288] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:13.583178 20986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:13.583732 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:13.577152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36341 (local address 127.0.0.1:38147)
0619 13:12:13.577361 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:13.577368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:13.577745 (+ 377us) server_negotiation.cc:407] Connection header received
0619 13:12:13.577955 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:13.577960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:13.578037 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:13.578161 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:13.578756 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:13.579708 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:13.581038 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:13.581504 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:13.581660 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:13.582881 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:13.582886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:13.582890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:13.582908 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:13.582945 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:13.582959 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:13.583102 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:13.583319 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 13:12:13.583389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:12:13.586355 20986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1257af6aeb1e4002b1d0a0233241f2a4, State: Running, Role: LEADER
I20260619 13:12:13.586971 20986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } }
I20260619 13:12:13.587713 20976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:13.595302 20987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } } }
I20260619 13:12:13.595269 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1257af6aeb1e4002b1d0a0233241f2a4. Latest consensus state: current_term: 1 leader_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1257af6aeb1e4002b1d0a0233241f2a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38147 } } }
I20260619 13:12:13.595757 20987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:13.595765 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4 [sys.catalog]: This master's current role is: LEADER
W20260619 13:12:13.601176 21000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:12:13.601325 21000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:12:13.601470 21001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:13.603220 21001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:13.611617 21001 catalog_manager.cc:1347] Generated new cluster ID: 185325240d0a4dffb86903fc47ff4e33
I20260619 13:12:13.611734 21001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:13.655272 21001 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:13.656605 21001 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:13.664741 21001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1257af6aeb1e4002b1d0a0233241f2a4: Generated new TSK 0
I20260619 13:12:13.665673 21001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:13.672917 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20909
W20260619 13:12:13.674347 20971 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54989
W20260619 13:12:13.675195 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 20909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:13.675401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 20909
W20260619 13:12:13.692879 20894 connection.cc:535] client connection to 127.0.0.1:38147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38147): Connection reset by peer (error 104)
2026-06-19T13:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:13 dist-test-slave-q11v krb5kdc[21007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:13 dist-test-slave-q11v krb5kdc[21007](info): set up 2 sockets
Jun 19 13:12:13 dist-test-slave-q11v krb5kdc[21007](info): commencing operation
krb5kdc: starting...
W20260619 13:12:14.786060 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:14 dist-test-slave-q11v krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:15.040473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:15.391355 21020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:15.391750 21020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:15.391933 21020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:15.399616 21020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:15.399797 21020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:15.399916 21020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:15.400038 21020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:15.400123 21020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:15.410722 21020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:15.412221 21020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:15.413647 21020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:15.426235 21025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:15.426312 21028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:15.426338 21026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:15.427847 21020 server_base.cc:1034] running on GCE node
I20260619 13:12:15.428655 21020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:15.430338 21020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:15.431572 21020 hybrid_clock.cc:648] HybridClock initialized: now 1781874735431536 us; error 54 us; skew 500 ppm
I20260619 13:12:15.431923 21020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:15 dist-test-slave-q11v krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:15.440284 21020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:15.442132 21020 webserver.cc:460] Webserver started at http://127.0.0.1:33941/ using document root <none> and password file <none>
I20260619 13:12:15.442963 21020 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:15.443094 21020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:15.443413 21020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:15.446590 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "83d5de54fd9845cb8e79a3b868c606a0"
format_stamp: "Formatted at 2026-06-19 13:12:15 on dist-test-slave-q11v"
server_key: "e904cf8b5f01933adb9220503f1e80db"
server_key_iv: "319f9c8c5b8004f7061054c42f915067"
server_key_version: "encryptionkey@0"
I20260619 13:12:15.447389 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "83d5de54fd9845cb8e79a3b868c606a0"
format_stamp: "Formatted at 2026-06-19 13:12:15 on dist-test-slave-q11v"
server_key: "e904cf8b5f01933adb9220503f1e80db"
server_key_iv: "319f9c8c5b8004f7061054c42f915067"
server_key_version: "encryptionkey@0"
I20260619 13:12:15.453078 21020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:12:15.456776 21035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:15.458343 21020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:12:15.458536 21020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "83d5de54fd9845cb8e79a3b868c606a0"
format_stamp: "Formatted at 2026-06-19 13:12:15 on dist-test-slave-q11v"
server_key: "e904cf8b5f01933adb9220503f1e80db"
server_key_iv: "319f9c8c5b8004f7061054c42f915067"
server_key_version: "encryptionkey@0"
I20260619 13:12:15.458752 21020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:15.464797 21020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:15.469209 21020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:15.469518 21020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:15.488812 21020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32989
I20260619 13:12:15.488853 21086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32989 every 8 connection(s)
I20260619 13:12:15.490365 21020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:15.494412 21087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:15.497555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21020
I20260619 13:12:15.497772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:15.498215 388 external_mini_cluster.cc:1428] Setting key c32ee5a1752bb910f1b80a7a1534aaf1
I20260619 13:12:15.506585 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:15.500315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38043 (local address 127.0.0.1:32989)
0619 13:12:15.501118 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:15.501131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:15.501172 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:12:15.503040 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:15.503062 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:15.503439 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:15.503795 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:15.504953 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:15.504959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:15.504972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:15.504985 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:15.505044 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:15.505065 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:15.505232 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:15.505380 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 13:12:15.505581 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":373,"threads_started":1}
I20260619 13:12:15.507571 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0: Bootstrap starting.
I20260619 13:12:15.510317 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:15.511256 21087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:15.512328 21087 log.cc:824] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:15.515755 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0: No bootstrap required, opened a new log
I20260619 13:12:15.517426 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:15.514681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32889 (local address 127.0.0.1:32989)
0619 13:12:15.514915 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:15.514924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:15.515378 (+ 454us) server_negotiation.cc:407] Connection header received
0619 13:12:15.515601 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:15.515605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:15.515653 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:15.515760 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:15.516752 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:15.516756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:15.516759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:15.516787 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:15.516825 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:15.516839 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:15.516930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:15.517071 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 13:12:15.517122 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:12:15.523743 21087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260619 13:12:15.524175 21087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:15.524304 21087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d5de54fd9845cb8e79a3b868c606a0, State: Initialized, Role: FOLLOWER
I20260619 13:12:15.524951 21087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260619 13:12:15.525213 21087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:15.525373 21087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:15.525553 21087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:15.527786 21087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260619 13:12:15.528308 21087 leader_election.cc:302] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d5de54fd9845cb8e79a3b868c606a0; no voters:
I20260619 13:12:15.529003 21087 leader_election.cc:288] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:15.529119 21097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:15.530028 21097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [term 1 LEADER]: Becoming Leader. State: Replica: 83d5de54fd9845cb8e79a3b868c606a0, State: Running, Role: LEADER
I20260619 13:12:15.530650 21097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260619 13:12:15.531589 21087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:15.533612 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d5de54fd9845cb8e79a3b868c606a0. Latest consensus state: current_term: 1 leader_uuid: "83d5de54fd9845cb8e79a3b868c606a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260619 13:12:15.534024 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:15.534067 21098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d5de54fd9845cb8e79a3b868c606a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d5de54fd9845cb8e79a3b868c606a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260619 13:12:15.534487 21098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:15.536057 21104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:15.541770 21104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:15.551065 21104 catalog_manager.cc:1347] Generated new cluster ID: 2a469f3800e044408aeb149abf00c69f
I20260619 13:12:15.551224 21104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:15.570241 21104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:15.570724 21104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:15.580984 21104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d5de54fd9845cb8e79a3b868c606a0: Generated new TSK 0
I20260619 13:12:15.581898 21104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:15.586524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21020
W20260619 13:12:15.587853 21082 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38043
W20260619 13:12:15.588562 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:15.588711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21020
2026-06-19T13:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:15 dist-test-slave-q11v krb5kdc[21118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:15 dist-test-slave-q11v krb5kdc[21118](info): set up 2 sockets
Jun 19 13:12:15 dist-test-slave-q11v krb5kdc[21118](info): commencing operation
krb5kdc: starting...
W20260619 13:12:16.694837 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:16 dist-test-slave-q11v krb5kdc[21118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:16.949298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:17.280773 21131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:17.281167 21131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:17.281334 21131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:17.288707 21131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:17.288878 21131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:17.288991 21131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:17.289101 21131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:17.289188 21131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:17.299273 21131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:17.300671 21131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:17.302040 21131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:17.314708 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:17.314829 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:17.315096 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:17.315876 21131 server_base.cc:1034] running on GCE node
I20260619 13:12:17.316766 21131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:17.318506 21131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:17.319777 21131 hybrid_clock.cc:648] HybridClock initialized: now 1781874737319739 us; error 48 us; skew 500 ppm
I20260619 13:12:17.320205 21131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:17 dist-test-slave-q11v krb5kdc[21118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:17.323560 21131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:17.325112 21131 webserver.cc:460] Webserver started at http://127.0.0.1:42001/ using document root <none> and password file <none>
I20260619 13:12:17.325897 21131 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:17.326042 21131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:17.326372 21131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:17.329378 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "603660947c7047958845d6261df042ee"
format_stamp: "Formatted at 2026-06-19 13:12:17 on dist-test-slave-q11v"
server_key: "941a165c988eee6b525b1a207e972498"
server_key_iv: "57a3398f3f1ee6e0e42189e44922f3e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:17.330226 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "603660947c7047958845d6261df042ee"
format_stamp: "Formatted at 2026-06-19 13:12:17 on dist-test-slave-q11v"
server_key: "941a165c988eee6b525b1a207e972498"
server_key_iv: "57a3398f3f1ee6e0e42189e44922f3e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:17.335371 21131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:12:17.339275 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:17.341058 21131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:12:17.341239 21131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "603660947c7047958845d6261df042ee"
format_stamp: "Formatted at 2026-06-19 13:12:17 on dist-test-slave-q11v"
server_key: "941a165c988eee6b525b1a207e972498"
server_key_iv: "57a3398f3f1ee6e0e42189e44922f3e1"
server_key_version: "encryptionkey@0"
I20260619 13:12:17.341451 21131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:17.364172 21131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:17.368111 21131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:17.368402 21131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:17.387169 21131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33555
I20260619 13:12:17.387297 21197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33555 every 8 connection(s)
I20260619 13:12:17.388582 21131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:17.392746 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:17.393805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21131
I20260619 13:12:17.393992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:17.394459 388 external_mini_cluster.cc:1428] Setting key be303c76b2a4c4417871300a54bd0eb2
I20260619 13:12:17.408577 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee: Bootstrap starting.
Jun 19 13:12:17 dist-test-slave-q11v krb5kdc[21118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:12:17.412325 21198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:17.413862 21198 log.cc:824] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:17.417706 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee: No bootstrap required, opened a new log
I20260619 13:12:17.421801 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:17.396566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58197 (local address 127.0.0.1:33555)
0619 13:12:17.397462 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:17.397480 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:17.397536 (+ 56us) server_negotiation.cc:407] Connection header received
0619 13:12:17.400050 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:17.400082 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:17.400678 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:17.401208 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:17.402610 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:17.404727 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:17.406200 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:17.407333 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:17.407568 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:17.412161 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:17.412176 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:17.412206 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:17.412269 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:17.416550 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:17.417461 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:17.417470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:17.417479 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:17.417760 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:17.418260 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:17.418265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:17.418268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:17.419009 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:17.419333 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:17.419790 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 13:12:17.420304 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":367,"threads_started":1}
I20260619 13:12:17.425820 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:17.427140 21198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260619 13:12:17.427508 21198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:17.427608 21198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603660947c7047958845d6261df042ee, State: Initialized, Role: FOLLOWER
I20260619 13:12:17.428234 21198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260619 13:12:17.428483 21198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:17.428632 21198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:17.428817 21198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:17.431622 21198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260619 13:12:17.432092 21198 leader_election.cc:302] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603660947c7047958845d6261df042ee; no voters:
I20260619 13:12:17.432690 21198 leader_election.cc:288] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:17.432789 21208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:17.434206 21208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [term 1 LEADER]: Becoming Leader. State: Replica: 603660947c7047958845d6261df042ee, State: Running, Role: LEADER
I20260619 13:12:17.434795 21208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260619 13:12:17.435345 21198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:17.437986 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603660947c7047958845d6261df042ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } } }
I20260619 13:12:17.438475 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:17.440754 21210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603660947c7047958845d6261df042ee. Latest consensus state: current_term: 1 leader_uuid: "603660947c7047958845d6261df042ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603660947c7047958845d6261df042ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } } }
I20260619 13:12:17.441138 21210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:17.443866 21213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:17.448361 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:17.430453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59339 (local address 127.0.0.1:33555)
0619 13:12:17.430635 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:17.430642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:17.430974 (+ 332us) server_negotiation.cc:407] Connection header received
0619 13:12:17.431106 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:17.431111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:17.431190 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:17.431301 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:17.432998 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:17.434255 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:17.435829 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:17.436285 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:17.436465 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:17.440164 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:17.440170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:17.440175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:17.440226 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:17.444369 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:17.445059 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:17.445066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:17.445069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:17.445219 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:17.445681 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:17.445684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:17.445686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:17.446154 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:17.447582 (+ 1428us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:17.447722 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 13:12:17.447957 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:12:17.450645 21213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:17.461566 21213 catalog_manager.cc:1347] Generated new cluster ID: fda7b1dcba474a1f86b7fb20f92f7381
I20260619 13:12:17.461719 21213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:17.497139 21213 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:17.499115 21213 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:17.511188 21213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603660947c7047958845d6261df042ee: Generated new TSK 0
I20260619 13:12:17.512358 21213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:17.523336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21131
I20260619 13:12:17.524834 21193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:17.618829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21131
2026-06-19T13:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:17 dist-test-slave-q11v krb5kdc[21230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:17 dist-test-slave-q11v krb5kdc[21230](info): set up 2 sockets
Jun 19 13:12:17 dist-test-slave-q11v krb5kdc[21230](info): commencing operation
krb5kdc: starting...
W20260619 13:12:18.722141 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:18 dist-test-slave-q11v krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:18.977008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:19.321344 21243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:19.321745 21243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:19.321920 21243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:19.329404 21243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:19.329573 21243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:19.329711 21243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:19.329829 21243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:19.329918 21243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:19.340022 21243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:19.341388 21243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:19.342765 21243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:19.354579 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:19.354631 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:19.354882 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:19.355648 21243 server_base.cc:1034] running on GCE node
I20260619 13:12:19.356465 21243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:19.357931 21243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:19.359166 21243 hybrid_clock.cc:648] HybridClock initialized: now 1781874739359133 us; error 56 us; skew 500 ppm
I20260619 13:12:19.359529 21243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:19 dist-test-slave-q11v krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:19.362738 21243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:19.364317 21243 webserver.cc:460] Webserver started at http://127.0.0.1:39429/ using document root <none> and password file <none>
I20260619 13:12:19.365010 21243 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:19.365157 21243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:19.365473 21243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:19.368517 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "bc7c2ce08c114e9b86847947f40d8c32"
format_stamp: "Formatted at 2026-06-19 13:12:19 on dist-test-slave-q11v"
server_key: "962b9da1a77593f350dce2762f0c2214"
server_key_iv: "5d7a6eef0a7828a1f3234d21908fa1bb"
server_key_version: "encryptionkey@0"
I20260619 13:12:19.369298 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc7c2ce08c114e9b86847947f40d8c32"
format_stamp: "Formatted at 2026-06-19 13:12:19 on dist-test-slave-q11v"
server_key: "962b9da1a77593f350dce2762f0c2214"
server_key_iv: "5d7a6eef0a7828a1f3234d21908fa1bb"
server_key_version: "encryptionkey@0"
I20260619 13:12:19.374593 21243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 13:12:19.378211 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:19.379725 21243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:12:19.379936 21243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "bc7c2ce08c114e9b86847947f40d8c32"
format_stamp: "Formatted at 2026-06-19 13:12:19 on dist-test-slave-q11v"
server_key: "962b9da1a77593f350dce2762f0c2214"
server_key_iv: "5d7a6eef0a7828a1f3234d21908fa1bb"
server_key_version: "encryptionkey@0"
I20260619 13:12:19.380144 21243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:19.402125 21243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:19.406406 21243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:19.406729 21243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:19.426116 21243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32935
I20260619 13:12:19.426154 21309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32935 every 8 connection(s)
I20260619 13:12:19.427577 21243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:19.431354 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:19.431821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21243
I20260619 13:12:19.432003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:19.432571 388 external_mini_cluster.cc:1428] Setting key bc01b78b8d5fb9d97af6c85c0526083e
I20260619 13:12:19.445209 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32: Bootstrap starting.
I20260619 13:12:19.446599 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:19.434487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56371 (local address 127.0.0.1:32935)
0619 13:12:19.435274 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:19.435286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:19.435327 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:12:19.437510 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:19.437533 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:19.438023 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:19.438368 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:19.438990 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:19.440268 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:19.442193 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:19.442926 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:19.443055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:19.444289 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:19.444297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:19.444312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:19.444856 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:19.444934 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:19.444958 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:19.445047 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:19.445296 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 13:12:19.445566 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":362,"threads_started":1}
I20260619 13:12:19.448508 21310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:19.449798 21310 log.cc:824] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:19.450291 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:19.452718 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32: No bootstrap required, opened a new log
I20260619 13:12:19.460824 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:19.454544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57447 (local address 127.0.0.1:32935)
0619 13:12:19.454694 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:19.454698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:19.455133 (+ 435us) server_negotiation.cc:407] Connection header received
0619 13:12:19.455385 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:19.455392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:19.455458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:19.455577 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:19.456035 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:19.456780 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:19.458219 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:19.458694 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:19.458844 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:19.459998 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:19.460003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:19.460006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:19.460027 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:19.460091 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:19.460113 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:19.460204 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:19.460419 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 13:12:19.460500 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:12:19.461488 21310 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } }
I20260619 13:12:19.461989 21310 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:19.462119 21310 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7c2ce08c114e9b86847947f40d8c32, State: Initialized, Role: FOLLOWER
I20260619 13:12:19.462728 21310 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } }
I20260619 13:12:19.462966 21310 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:19.463125 21310 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:19.463299 21310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:19.465525 21310 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } }
I20260619 13:12:19.466121 21310 leader_election.cc:302] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7c2ce08c114e9b86847947f40d8c32; no voters:
I20260619 13:12:19.466737 21310 leader_election.cc:288] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:19.466950 21320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:19.468165 21320 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [term 1 LEADER]: Becoming Leader. State: Replica: bc7c2ce08c114e9b86847947f40d8c32, State: Running, Role: LEADER
I20260619 13:12:19.468743 21320 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } }
I20260619 13:12:19.469027 21310 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:19.471885 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7c2ce08c114e9b86847947f40d8c32. Latest consensus state: current_term: 1 leader_uuid: "bc7c2ce08c114e9b86847947f40d8c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } } }
I20260619 13:12:19.472324 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:19.471853 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7c2ce08c114e9b86847947f40d8c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7c2ce08c114e9b86847947f40d8c32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32935 } } }
I20260619 13:12:19.472755 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:19.473246 21326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:19.479733 21326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:19.491518 21326 catalog_manager.cc:1347] Generated new cluster ID: 577c9b7a32c0452ca24dcf27de82a20b
I20260619 13:12:19.491652 21326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:19.502104 21326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:19.503579 21326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:19.521345 21326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7c2ce08c114e9b86847947f40d8c32: Generated new TSK 0
I20260619 13:12:19.522280 21326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:19.531908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21243
W20260619 13:12:19.533047 21305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56371
W20260619 13:12:19.533757 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21243: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:19.533931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21243
2026-06-19T13:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:19 dist-test-slave-q11v krb5kdc[21341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:19 dist-test-slave-q11v krb5kdc[21341](info): set up 2 sockets
Jun 19 13:12:19 dist-test-slave-q11v krb5kdc[21341](info): commencing operation
krb5kdc: starting...
W20260619 13:12:20.654003 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:20 dist-test-slave-q11v krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:20.919343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:21.280459 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:21.280857 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:21.281059 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:21.288803 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:21.289000 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:21.289139 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:21.289255 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:21.289352 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:21.299952 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:21.301420 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:21.302928 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:21.315652 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:21.315711 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:21.315747 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:21.317011 21354 server_base.cc:1034] running on GCE node
I20260619 13:12:21.318022 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:21.319618 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:21.320878 21354 hybrid_clock.cc:648] HybridClock initialized: now 1781874741320845 us; error 72 us; skew 500 ppm
I20260619 13:12:21.321262 21354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:21 dist-test-slave-q11v krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:21.330365 21354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:21.332242 21354 webserver.cc:460] Webserver started at http://127.0.0.1:34047/ using document root <none> and password file <none>
I20260619 13:12:21.333040 21354 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:21.333205 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:21.333557 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:21.337097 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "de4c0e6f1bc74ffba4a755646e6d352a"
format_stamp: "Formatted at 2026-06-19 13:12:21 on dist-test-slave-q11v"
server_key: "60ca583aedc6774a344fe8eb8f9844cc"
server_key_iv: "1e54330ef204a53366c77aecb6cbdda6"
server_key_version: "encryptionkey@0"
I20260619 13:12:21.338150 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "de4c0e6f1bc74ffba4a755646e6d352a"
format_stamp: "Formatted at 2026-06-19 13:12:21 on dist-test-slave-q11v"
server_key: "60ca583aedc6774a344fe8eb8f9844cc"
server_key_iv: "1e54330ef204a53366c77aecb6cbdda6"
server_key_version: "encryptionkey@0"
I20260619 13:12:21.343544 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:12:21.347404 21369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:21.348889 21354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:12:21.349045 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "de4c0e6f1bc74ffba4a755646e6d352a"
format_stamp: "Formatted at 2026-06-19 13:12:21 on dist-test-slave-q11v"
server_key: "60ca583aedc6774a344fe8eb8f9844cc"
server_key_iv: "1e54330ef204a53366c77aecb6cbdda6"
server_key_version: "encryptionkey@0"
I20260619 13:12:21.349221 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:21.355050 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:21.359287 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:21.359552 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:21.378594 21354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46561
I20260619 13:12:21.378634 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46561 every 8 connection(s)
I20260619 13:12:21.380043 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:21.383873 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:21.386049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21354
I20260619 13:12:21.386252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:21.386821 388 external_mini_cluster.cc:1428] Setting key 4ae07210c7ec5d601e65c2c1a5b26ee6
I20260619 13:12:21.395419 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:21.388805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50323 (local address 127.0.0.1:46561)
0619 13:12:21.389751 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:21.389764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:21.389814 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:12:21.391523 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:21.391550 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:21.391919 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:21.392331 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:21.393376 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:21.393382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:21.393396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:21.393411 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:21.393477 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:21.393497 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:21.393798 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:21.393941 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 13:12:21.394140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":367,"threads_started":1}
I20260619 13:12:21.398648 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a: Bootstrap starting.
I20260619 13:12:21.399860 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:21.402356 21421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:21.403622 21421 log.cc:824] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:21.407234 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a: No bootstrap required, opened a new log
I20260619 13:12:21.407531 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:21.404762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40507 (local address 127.0.0.1:46561)
0619 13:12:21.404967 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:21.404975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:21.405369 (+ 394us) server_negotiation.cc:407] Connection header received
0619 13:12:21.405522 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:21.405528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:21.405629 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:21.405740 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:21.406773 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:21.406778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:21.406782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:21.406800 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:21.406846 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:21.406866 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:21.406992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:21.407156 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 13:12:21.407226 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:12:21.415387 21421 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260619 13:12:21.415838 21421 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:21.415943 21421 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4c0e6f1bc74ffba4a755646e6d352a, State: Initialized, Role: FOLLOWER
I20260619 13:12:21.416592 21421 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260619 13:12:21.416846 21421 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:21.417013 21421 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:21.417181 21421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:21.419916 21421 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260619 13:12:21.420482 21421 leader_election.cc:302] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4c0e6f1bc74ffba4a755646e6d352a; no voters:
I20260619 13:12:21.421093 21421 leader_election.cc:288] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:21.421278 21431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:21.422039 21431 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [term 1 LEADER]: Becoming Leader. State: Replica: de4c0e6f1bc74ffba4a755646e6d352a, State: Running, Role: LEADER
I20260619 13:12:21.422626 21431 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260619 13:12:21.423309 21421 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:21.426023 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260619 13:12:21.425794 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4c0e6f1bc74ffba4a755646e6d352a. Latest consensus state: current_term: 1 leader_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c0e6f1bc74ffba4a755646e6d352a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260619 13:12:21.426342 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:21.426342 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:21.427423 21440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:21.433249 21440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:21.442914 21440 catalog_manager.cc:1347] Generated new cluster ID: 7f45b0ab6b744dfebb6df10993c3dda1
I20260619 13:12:21.443053 21440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:21.479380 21440 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:21.479844 21440 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:21.487517 21440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4c0e6f1bc74ffba4a755646e6d352a: Generated new TSK 0
I20260619 13:12:21.488386 21440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:21.495651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21354
W20260619 13:12:21.496752 21416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50323
W20260619 13:12:21.497478 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:21.497658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21354
2026-06-19T13:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:21 dist-test-slave-q11v krb5kdc[21452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:21 dist-test-slave-q11v krb5kdc[21452](info): set up 2 sockets
Jun 19 13:12:21 dist-test-slave-q11v krb5kdc[21452](info): commencing operation
krb5kdc: starting...
W20260619 13:12:22.614197 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:22 dist-test-slave-q11v krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:22.882042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:23.223023 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:23.223542 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:23.223731 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:23.231300 21465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:23.231467 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:23.231585 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:23.231701 21465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:23.231796 21465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:23.242077 21465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:23.243515 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:23.244992 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:23.257014 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:23.256982 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:23.257041 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:23.258232 21465 server_base.cc:1034] running on GCE node
I20260619 13:12:23.259093 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:23.260615 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:23.261879 21465 hybrid_clock.cc:648] HybridClock initialized: now 1781874743261817 us; error 44 us; skew 500 ppm
I20260619 13:12:23.262328 21465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:23 dist-test-slave-q11v krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:23.265806 21465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:23.267452 21465 webserver.cc:460] Webserver started at http://127.0.0.1:33811/ using document root <none> and password file <none>
I20260619 13:12:23.268204 21465 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:23.268355 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:23.268685 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:23.271952 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "3cdc125a37a84961a453f6c56907e5ee"
format_stamp: "Formatted at 2026-06-19 13:12:23 on dist-test-slave-q11v"
server_key: "d5e33cf3f8d7472c19b98b79b5237659"
server_key_iv: "725aa6eded9bd237f1b1d98b5ce22d2d"
server_key_version: "encryptionkey@0"
I20260619 13:12:23.272786 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "3cdc125a37a84961a453f6c56907e5ee"
format_stamp: "Formatted at 2026-06-19 13:12:23 on dist-test-slave-q11v"
server_key: "d5e33cf3f8d7472c19b98b79b5237659"
server_key_iv: "725aa6eded9bd237f1b1d98b5ce22d2d"
server_key_version: "encryptionkey@0"
I20260619 13:12:23.278194 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:12:23.282061 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:23.283507 21465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:23.283731 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "3cdc125a37a84961a453f6c56907e5ee"
format_stamp: "Formatted at 2026-06-19 13:12:23 on dist-test-slave-q11v"
server_key: "d5e33cf3f8d7472c19b98b79b5237659"
server_key_iv: "725aa6eded9bd237f1b1d98b5ce22d2d"
server_key_version: "encryptionkey@0"
I20260619 13:12:23.283952 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:23.308480 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:23.313251 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:23.313575 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:23.333298 21465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36251
I20260619 13:12:23.333321 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36251 every 8 connection(s)
I20260619 13:12:23.334854 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:23.338897 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:23.339365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21465
I20260619 13:12:23.339548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:23.339998 388 external_mini_cluster.cc:1428] Setting key ffc916d9d2fd6d063393a1539f095c73
I20260619 13:12:23.354008 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee: Bootstrap starting.
Jun 19 13:12:23 dist-test-slave-q11v krb5kdc[21452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:12:23.357857 21532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:23.359309 21532 log.cc:824] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:23.362920 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee: No bootstrap required, opened a new log
I20260619 13:12:23.365260 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:23.341829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:56507 (local address 127.0.0.1:36251)
0619 13:12:23.342688 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:23.342700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:23.342745 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:12:23.344299 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:23.344322 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:23.344752 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:23.345089 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:23.347028 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:23.348364 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:23.350219 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:23.350924 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:23.351064 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:23.356062 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:23.356076 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:23.356096 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:23.356142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:23.360229 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:23.361256 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:23.361267 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:23.361278 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:23.361514 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:23.362028 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:23.362034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:23.362037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:23.362721 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:23.363054 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:23.363537 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 13:12:23.363903 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":425,"threads_started":1}
I20260619 13:12:23.369366 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:23.372880 21532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260619 13:12:23.373332 21532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:23.373433 21532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdc125a37a84961a453f6c56907e5ee, State: Initialized, Role: FOLLOWER
I20260619 13:12:23.374146 21532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260619 13:12:23.374481 21532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:23.374643 21532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:23.374871 21532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:23.377467 21532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260619 13:12:23.378015 21532 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdc125a37a84961a453f6c56907e5ee; no voters:
I20260619 13:12:23.378690 21532 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:23.378780 21542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:23.379463 21542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [term 1 LEADER]: Becoming Leader. State: Replica: 3cdc125a37a84961a453f6c56907e5ee, State: Running, Role: LEADER
I20260619 13:12:23.380311 21542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260619 13:12:23.383329 21532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:23.385464 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdc125a37a84961a453f6c56907e5ee. Latest consensus state: current_term: 1 leader_uuid: "3cdc125a37a84961a453f6c56907e5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } } }
I20260619 13:12:23.385937 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:23.387943 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdc125a37a84961a453f6c56907e5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdc125a37a84961a453f6c56907e5ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } } }
I20260619 13:12:23.393705 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:23.394816 21550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:23.395738 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:23.374195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:58301 (local address 127.0.0.1:36251)
0619 13:12:23.374405 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:23.374416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:23.374973 (+ 557us) server_negotiation.cc:407] Connection header received
0619 13:12:23.375230 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:23.375237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:23.375313 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:23.375417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:23.377001 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:23.377880 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:23.379388 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:23.380069 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:23.380153 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:23.383423 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:23.383429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:23.383434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:23.383474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:23.390902 (+ 7428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:23.392685 (+ 1783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:23.392709 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:23.392714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:23.392935 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:23.393695 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:23.393712 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:23.393715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:23.394237 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:23.394562 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:23.395171 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 13:12:23.395365 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:12:23.402782 21550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:23.413503 21550 catalog_manager.cc:1347] Generated new cluster ID: 5cbe8f3f6a3a41d2ba5faf697865ea9b
I20260619 13:12:23.413770 21550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:23.426630 21550 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:23.428553 21550 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:23.441968 21550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdc125a37a84961a453f6c56907e5ee: Generated new TSK 0
I20260619 13:12:23.442907 21550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:23.456970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21465
I20260619 13:12:23.458460 21527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:23.554222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21465
W20260619 13:12:23.571846 21450 connection.cc:535] client connection to 127.0.0.1:36251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36251): Connection reset by peer (error 104)
2026-06-19T13:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:23 dist-test-slave-q11v krb5kdc[21564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:23 dist-test-slave-q11v krb5kdc[21564](info): set up 2 sockets
Jun 19 13:12:23 dist-test-slave-q11v krb5kdc[21564](info): commencing operation
krb5kdc: starting...
W20260619 13:12:24.670179 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:24 dist-test-slave-q11v krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:24.933105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:25.276178 21577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:25.276561 21577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:25.276734 21577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:25.284000 21577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:25.284169 21577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:25.284281 21577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:25.284396 21577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:25.284484 21577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:25.294574 21577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:25.295939 21577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:25.297320 21577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:25.309556 21583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:25.309680 21585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:25.309621 21582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:25.310467 21577 server_base.cc:1034] running on GCE node
I20260619 13:12:25.311234 21577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:25.312614 21577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:25.313844 21577 hybrid_clock.cc:648] HybridClock initialized: now 1781874745313808 us; error 51 us; skew 500 ppm
I20260619 13:12:25.314203 21577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:25 dist-test-slave-q11v krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:25.317335 21577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:25.319029 21577 webserver.cc:460] Webserver started at http://127.0.0.1:35563/ using document root <none> and password file <none>
I20260619 13:12:25.319774 21577 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:25.319904 21577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:25.320251 21577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:25.323434 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "33fe06551d0a4441a04eb02fc853dc02"
format_stamp: "Formatted at 2026-06-19 13:12:25 on dist-test-slave-q11v"
server_key: "36f79660708686eaaf8287c733eaf251"
server_key_iv: "98af392196a3558d49b70bc6ba8115a3"
server_key_version: "encryptionkey@0"
I20260619 13:12:25.324239 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "33fe06551d0a4441a04eb02fc853dc02"
format_stamp: "Formatted at 2026-06-19 13:12:25 on dist-test-slave-q11v"
server_key: "36f79660708686eaaf8287c733eaf251"
server_key_iv: "98af392196a3558d49b70bc6ba8115a3"
server_key_version: "encryptionkey@0"
I20260619 13:12:25.329727 21577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:12:25.333184 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:25.334622 21577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:25.334808 21577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "33fe06551d0a4441a04eb02fc853dc02"
format_stamp: "Formatted at 2026-06-19 13:12:25 on dist-test-slave-q11v"
server_key: "36f79660708686eaaf8287c733eaf251"
server_key_iv: "98af392196a3558d49b70bc6ba8115a3"
server_key_version: "encryptionkey@0"
I20260619 13:12:25.335008 21577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:25.368634 21577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:25.372922 21577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:25.373235 21577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:25.392912 21577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42961
I20260619 13:12:25.392966 21643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42961 every 8 connection(s)
I20260619 13:12:25.394460 21577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:25.398702 21644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:25.399163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21577
I20260619 13:12:25.399344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:25.399775 388 external_mini_cluster.cc:1428] Setting key 1cddbc4a5aacacc085a8aded19c0d87b
I20260619 13:12:25.412978 21644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02: Bootstrap starting.
I20260619 13:12:25.413942 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:25.401583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:58263 (local address 127.0.0.1:42961)
0619 13:12:25.402487 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:25.402500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:25.402548 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:12:25.404412 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:25.404437 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:25.404910 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:25.405383 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:25.406023 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:25.407454 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:25.408948 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:25.409807 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:25.410034 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:25.411482 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:25.411490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:25.411505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:25.412111 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:25.412189 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:25.412216 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:25.412383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:25.412673 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 13:12:25.412845 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":357,"threads_started":1}
I20260619 13:12:25.416553 21644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:25.417923 21644 log.cc:824] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:25.418057 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:25.421002 21644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02: No bootstrap required, opened a new log
I20260619 13:12:25.431119 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:25.422553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:34165 (local address 127.0.0.1:42961)
0619 13:12:25.422771 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:25.422777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:25.423197 (+ 420us) server_negotiation.cc:407] Connection header received
0619 13:12:25.423391 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:25.423395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:25.423463 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:25.423547 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:25.424245 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:25.425148 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:25.427316 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:25.428277 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:25.428490 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:25.430207 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:25.430214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:25.430220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:25.430244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:25.430296 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:25.430320 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:25.430452 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:25.430646 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 13:12:25.430726 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:12:25.432936 21644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260619 13:12:25.433357 21644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:25.433439 21644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fe06551d0a4441a04eb02fc853dc02, State: Initialized, Role: FOLLOWER
I20260619 13:12:25.434278 21644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260619 13:12:25.434514 21644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:25.434624 21644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:25.434759 21644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:25.437705 21644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260619 13:12:25.438362 21644 leader_election.cc:302] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fe06551d0a4441a04eb02fc853dc02; no voters:
I20260619 13:12:25.439087 21644 leader_election.cc:288] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:25.439307 21654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:25.440119 21654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [term 1 LEADER]: Becoming Leader. State: Replica: 33fe06551d0a4441a04eb02fc853dc02, State: Running, Role: LEADER
I20260619 13:12:25.440758 21654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } }
I20260619 13:12:25.441284 21644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:25.443766 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fe06551d0a4441a04eb02fc853dc02. Latest consensus state: current_term: 1 leader_uuid: "33fe06551d0a4441a04eb02fc853dc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } } }
I20260619 13:12:25.444295 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:25.445194 21663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:25.445166 21655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fe06551d0a4441a04eb02fc853dc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe06551d0a4441a04eb02fc853dc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42961 } } }
I20260619 13:12:25.447225 21655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:25.450383 21663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:25.462432 21663 catalog_manager.cc:1347] Generated new cluster ID: 5b7a294ffb2e44ada13a9104f8d10f28
I20260619 13:12:25.462584 21663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:25.482967 21663 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:25.484402 21663 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:25.496544 21663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fe06551d0a4441a04eb02fc853dc02: Generated new TSK 0
I20260619 13:12:25.497491 21663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:25.506068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21577
W20260619 13:12:25.507344 21639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:58263
W20260619 13:12:25.508183 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:25.508359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21577
W20260619 13:12:25.525979 21562 connection.cc:535] client connection to 127.0.0.1:42961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42961): Connection reset by peer (error 104)
2026-06-19T13:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:25 dist-test-slave-q11v krb5kdc[21675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:25 dist-test-slave-q11v krb5kdc[21675](info): set up 2 sockets
Jun 19 13:12:25 dist-test-slave-q11v krb5kdc[21675](info): commencing operation
krb5kdc: starting...
W20260619 13:12:26.626135 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:26 dist-test-slave-q11v krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:26.878441 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:27.209583 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:27.209956 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:27.210129 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:27.217396 21688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:27.217509 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:27.217587 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:27.217765 21688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:27.217820 21688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:27.227777 21688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:27.229115 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:27.230484 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:27.242296 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:27.242350 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:27.242328 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:27.243597 21688 server_base.cc:1034] running on GCE node
I20260619 13:12:27.244485 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:27.245985 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:27.247184 21688 hybrid_clock.cc:648] HybridClock initialized: now 1781874747247144 us; error 31 us; skew 500 ppm
I20260619 13:12:27.247550 21688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:27 dist-test-slave-q11v krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:27.255587 21688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:27.257174 21688 webserver.cc:460] Webserver started at http://127.0.0.1:35019/ using document root <none> and password file <none>
I20260619 13:12:27.257980 21688 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:27.258109 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:27.258440 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:27.261456 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "c89a8d89b2ab42e0854846bab9ab83e9"
format_stamp: "Formatted at 2026-06-19 13:12:27 on dist-test-slave-q11v"
server_key: "cdfc61e2b3a2ba5fb013109928923c40"
server_key_iv: "ef8a63f60fcd84ca21d613eb7b4a0ab1"
server_key_version: "encryptionkey@0"
I20260619 13:12:27.262359 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "c89a8d89b2ab42e0854846bab9ab83e9"
format_stamp: "Formatted at 2026-06-19 13:12:27 on dist-test-slave-q11v"
server_key: "cdfc61e2b3a2ba5fb013109928923c40"
server_key_iv: "ef8a63f60fcd84ca21d613eb7b4a0ab1"
server_key_version: "encryptionkey@0"
I20260619 13:12:27.267624 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:12:27.271193 21703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:27.272545 21688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:12:27.272725 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "c89a8d89b2ab42e0854846bab9ab83e9"
format_stamp: "Formatted at 2026-06-19 13:12:27 on dist-test-slave-q11v"
server_key: "cdfc61e2b3a2ba5fb013109928923c40"
server_key_iv: "ef8a63f60fcd84ca21d613eb7b4a0ab1"
server_key_version: "encryptionkey@0"
I20260619 13:12:27.272924 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:27.278649 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:27.283281 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:27.283572 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:27.302491 21688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42723
I20260619 13:12:27.302541 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42723 every 8 connection(s)
I20260619 13:12:27.303925 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:27.307672 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:27.313314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21688
I20260619 13:12:27.313499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:27.313979 388 external_mini_cluster.cc:1428] Setting key e7d64bc8998890759a393ab302b8166a
I20260619 13:12:27.318590 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9: Bootstrap starting.
I20260619 13:12:27.322106 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:27.315772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:41839 (local address 127.0.0.1:42723)
0619 13:12:27.316632 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:27.316645 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:27.316695 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:12:27.318452 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:27.318475 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:27.318905 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:27.319251 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:27.320438 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:27.320445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:27.320457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:27.320473 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:27.320542 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:27.320565 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:27.320750 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:27.320856 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 13:12:27.321066 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":404,"threads_started":1}
I20260619 13:12:27.322461 21755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:27.323525 21755 log.cc:824] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:27.326287 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:27.326776 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9: No bootstrap required, opened a new log
I20260619 13:12:27.333629 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:27.330889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:40679 (local address 127.0.0.1:42723)
0619 13:12:27.331094 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:27.331103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:27.331467 (+ 364us) server_negotiation.cc:407] Connection header received
0619 13:12:27.331670 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:27.331675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:27.331738 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:27.331855 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:27.332823 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:27.332838 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:27.332844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:27.332860 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:27.332906 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:27.332921 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:27.333054 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:27.333171 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:12:27.333242 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:12:27.335892 21755 raft_consensus.cc:348] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260619 13:12:27.336288 21755 raft_consensus.cc:374] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:27.336412 21755 raft_consensus.cc:729] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89a8d89b2ab42e0854846bab9ab83e9, State: Initialized, Role: FOLLOWER
I20260619 13:12:27.337009 21755 consensus_queue.cc:260] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260619 13:12:27.337242 21755 raft_consensus.cc:388] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:27.337400 21755 raft_consensus.cc:482] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:27.337580 21755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:27.340106 21755 raft_consensus.cc:504] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260619 13:12:27.340729 21755 leader_election.cc:302] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c89a8d89b2ab42e0854846bab9ab83e9; no voters:
I20260619 13:12:27.341243 21755 leader_election.cc:288] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:27.341367 21765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:27.342116 21765 raft_consensus.cc:686] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [term 1 LEADER]: Becoming Leader. State: Replica: c89a8d89b2ab42e0854846bab9ab83e9, State: Running, Role: LEADER
I20260619 13:12:27.342598 21765 consensus_queue.cc:237] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260619 13:12:27.343364 21755 sys_catalog.cc:564] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:27.345492 21767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c89a8d89b2ab42e0854846bab9ab83e9. Latest consensus state: current_term: 1 leader_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260619 13:12:27.345858 21766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89a8d89b2ab42e0854846bab9ab83e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260619 13:12:27.346161 21766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:27.346038 21767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:27.348409 21773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:27.354007 21773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:27.365010 21773 catalog_manager.cc:1347] Generated new cluster ID: 92247931dacf4bc29a92e10ae173f3f5
I20260619 13:12:27.365149 21773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:27.381266 21773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:27.381776 21773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:27.388209 21773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c89a8d89b2ab42e0854846bab9ab83e9: Generated new TSK 0
I20260619 13:12:27.389041 21773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:27.395505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21688
W20260619 13:12:27.396596 21750 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:41839
W20260619 13:12:27.397310 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21688: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:27.397473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21688
2026-06-19T13:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:27 dist-test-slave-q11v krb5kdc[21786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:27 dist-test-slave-q11v krb5kdc[21786](info): set up 2 sockets
Jun 19 13:12:27 dist-test-slave-q11v krb5kdc[21786](info): commencing operation
krb5kdc: starting...
W20260619 13:12:28.518182 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:28 dist-test-slave-q11v krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:28.784761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:29.126291 21799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:29.126607 21799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:29.126818 21799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:29.134479 21799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:29.134635 21799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:29.134732 21799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:29.134859 21799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:29.134951 21799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:29.145378 21799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:29.146824 21799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:29.148267 21799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:29.160759 21804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:29.160768 21805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:29.160744 21807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:29.161669 21799 server_base.cc:1034] running on GCE node
I20260619 13:12:29.162390 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:29.163743 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:29.164942 21799 hybrid_clock.cc:648] HybridClock initialized: now 1781874749164900 us; error 36 us; skew 500 ppm
I20260619 13:12:29.165313 21799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:29 dist-test-slave-q11v krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:29.168480 21799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:29.170104 21799 webserver.cc:460] Webserver started at http://127.0.0.1:34787/ using document root <none> and password file <none>
I20260619 13:12:29.170806 21799 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:29.170953 21799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:29.171309 21799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:29.175866 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "022bf8d007e44e708d044c2c287eca07"
format_stamp: "Formatted at 2026-06-19 13:12:29 on dist-test-slave-q11v"
server_key: "316af64dac25a5756006ae007101199b"
server_key_iv: "737df8d288a75e6fbdd067c45579b631"
server_key_version: "encryptionkey@0"
I20260619 13:12:29.176994 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "022bf8d007e44e708d044c2c287eca07"
format_stamp: "Formatted at 2026-06-19 13:12:29 on dist-test-slave-q11v"
server_key: "316af64dac25a5756006ae007101199b"
server_key_iv: "737df8d288a75e6fbdd067c45579b631"
server_key_version: "encryptionkey@0"
I20260619 13:12:29.184171 21799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260619 13:12:29.189344 21814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:29.191459 21799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:12:29.191648 21799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "022bf8d007e44e708d044c2c287eca07"
format_stamp: "Formatted at 2026-06-19 13:12:29 on dist-test-slave-q11v"
server_key: "316af64dac25a5756006ae007101199b"
server_key_iv: "737df8d288a75e6fbdd067c45579b631"
server_key_version: "encryptionkey@0"
I20260619 13:12:29.191912 21799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:29.215766 21799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:29.221573 21799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:29.221941 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:29.241474 21799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45589
I20260619 13:12:29.241506 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45589 every 8 connection(s)
I20260619 13:12:29.243079 21799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:29.247027 21866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:29.250905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21799
I20260619 13:12:29.251093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:29.251647 388 external_mini_cluster.cc:1428] Setting key 1b40dc67860f8f5f4a2c842a5b2b33b1
I20260619 13:12:29.260023 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07: Bootstrap starting.
I20260619 13:12:29.263213 21866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:29.264300 21866 log.cc:824] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:12:29 dist-test-slave-q11v krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:12:29.267720 21866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07: No bootstrap required, opened a new log
I20260619 13:12:29.276520 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:29.253795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:34651 (local address 127.0.0.1:45589)
0619 13:12:29.254726 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:29.254739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:29.254781 (+ 42us) server_negotiation.cc:407] Connection header received
0619 13:12:29.256475 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:29.256500 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:29.256973 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:29.257450 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:29.258984 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:29.260376 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:29.261755 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:29.262451 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:29.262651 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:29.267083 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:29.267095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:29.267115 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:29.267162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:29.271395 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:29.272342 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:29.272355 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:29.272365 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:29.272582 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:29.273047 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:29.273054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:29.273063 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:29.273864 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:29.274309 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:29.274698 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 13:12:29.275083 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":488,"threads_started":1}
I20260619 13:12:29.278801 21866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260619 13:12:29.279296 21866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:29.279428 21866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022bf8d007e44e708d044c2c287eca07, State: Initialized, Role: FOLLOWER
I20260619 13:12:29.280063 21866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260619 13:12:29.280378 21866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:29.280535 21866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:29.280714 21866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:29.281101 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:29.283041 21866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260619 13:12:29.283538 21866 leader_election.cc:302] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 022bf8d007e44e708d044c2c287eca07; no voters:
I20260619 13:12:29.284201 21866 leader_election.cc:288] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:29.284314 21873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:29.287082 21866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:29.291560 21873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [term 1 LEADER]: Becoming Leader. State: Replica: 022bf8d007e44e708d044c2c287eca07, State: Running, Role: LEADER
I20260619 13:12:29.292346 21873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260619 13:12:29.295706 21876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "022bf8d007e44e708d044c2c287eca07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } } }
I20260619 13:12:29.296185 21876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:29.296804 21877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022bf8d007e44e708d044c2c287eca07. Latest consensus state: current_term: 1 leader_uuid: "022bf8d007e44e708d044c2c287eca07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022bf8d007e44e708d044c2c287eca07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } } }
I20260619 13:12:29.297130 21877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:29.300891 21886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:29.307027 21886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:29.311977 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:29.285972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:57429 (local address 127.0.0.1:45589)
0619 13:12:29.286149 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:29.286157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:29.292413 (+ 6256us) server_negotiation.cc:407] Connection header received
0619 13:12:29.292638 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:29.292646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:29.292737 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:29.292858 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:29.295186 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:29.296368 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:29.298908 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:29.299844 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:29.299988 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:29.303626 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:29.303633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:29.303638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:29.303699 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:29.307648 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:29.308754 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:29.308759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:29.308763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:29.308910 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:29.309488 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:29.309493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:29.309495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:29.309944 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:29.310225 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:29.311424 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0619 13:12:29.311621 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:12:29.316985 21886 catalog_manager.cc:1347] Generated new cluster ID: 561b8044aee545c39a301450b8cc03e9
I20260619 13:12:29.317103 21886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:29.352370 21886 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:29.354405 21886 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:29.365068 21886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022bf8d007e44e708d044c2c287eca07: Generated new TSK 0
I20260619 13:12:29.366206 21886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:29.380966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21799
I20260619 13:12:29.383224 21861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:29.480436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21799
W20260619 13:12:29.498184 21784 connection.cc:535] client connection to 127.0.0.1:45589 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45589): Connection reset by peer (error 104)
2026-06-19T13:12:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:29 dist-test-slave-q11v krb5kdc[21899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:29 dist-test-slave-q11v krb5kdc[21899](info): set up 2 sockets
Jun 19 13:12:29 dist-test-slave-q11v krb5kdc[21899](info): commencing operation
krb5kdc: starting...
W20260619 13:12:30.586059 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:30 dist-test-slave-q11v krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:30.852965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:31.194396 21912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:31.194741 21912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:31.194917 21912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:31.202402 21912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:31.202533 21912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:31.202615 21912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:31.202720 21912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:31.202792 21912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:31.212922 21912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:31.214370 21912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:31.215745 21912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:31.228197 21920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:31.228163 21918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:31.228724 21917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:31.229239 21912 server_base.cc:1034] running on GCE node
I20260619 13:12:31.230196 21912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:31.231756 21912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:31.232991 21912 hybrid_clock.cc:648] HybridClock initialized: now 1781874751232908 us; error 74 us; skew 500 ppm
I20260619 13:12:31.233458 21912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:31 dist-test-slave-q11v krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:31.236907 21912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:31.238778 21912 webserver.cc:460] Webserver started at http://127.0.0.1:37353/ using document root <none> and password file <none>
I20260619 13:12:31.239573 21912 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:31.239683 21912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:31.240053 21912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:31.243366 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "ae4c04b72002443d8b20d94e800a96c9"
format_stamp: "Formatted at 2026-06-19 13:12:31 on dist-test-slave-q11v"
server_key: "b8a813efb03a4aafa5897d87aa52af3e"
server_key_iv: "328841599734c64e343954c8a2cc6c49"
server_key_version: "encryptionkey@0"
I20260619 13:12:31.244253 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "ae4c04b72002443d8b20d94e800a96c9"
format_stamp: "Formatted at 2026-06-19 13:12:31 on dist-test-slave-q11v"
server_key: "b8a813efb03a4aafa5897d87aa52af3e"
server_key_iv: "328841599734c64e343954c8a2cc6c49"
server_key_version: "encryptionkey@0"
I20260619 13:12:31.249811 21912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:12:31.253541 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:31.255084 21912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:31.255296 21912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "ae4c04b72002443d8b20d94e800a96c9"
format_stamp: "Formatted at 2026-06-19 13:12:31 on dist-test-slave-q11v"
server_key: "b8a813efb03a4aafa5897d87aa52af3e"
server_key_iv: "328841599734c64e343954c8a2cc6c49"
server_key_version: "encryptionkey@0"
I20260619 13:12:31.255520 21912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:31.271966 21912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:31.276328 21912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:31.276635 21912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:31.296762 21912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32877
I20260619 13:12:31.296793 21978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32877 every 8 connection(s)
I20260619 13:12:31.298306 21912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:31.299404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 21912
I20260619 13:12:31.299582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:31.300036 388 external_mini_cluster.cc:1428] Setting key 928239c59a1060858fa357ad80788514
I20260619 13:12:31.303180 21979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:31.314694 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:31.302146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:55353 (local address 127.0.0.1:32877)
0619 13:12:31.302928 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:31.302941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:31.302982 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:12:31.304739 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:31.304763 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:31.305184 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:31.305515 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:31.306193 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:31.307573 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:31.309518 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:31.310328 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:31.310471 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:31.311915 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:31.311923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:31.311949 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:31.312584 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:31.312670 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:31.312700 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:31.312842 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:31.313284 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 13:12:31.313503 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":359,"threads_started":1}
I20260619 13:12:31.316200 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9: Bootstrap starting.
I20260619 13:12:31.318965 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:31.319823 21979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:31.320900 21979 log.cc:824] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:31.324338 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9: No bootstrap required, opened a new log
I20260619 13:12:31.331086 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:31.323472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:53475 (local address 127.0.0.1:32877)
0619 13:12:31.323678 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:31.323686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:31.324257 (+ 571us) server_negotiation.cc:407] Connection header received
0619 13:12:31.324502 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:31.324514 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:31.324584 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:31.324679 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:31.325411 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:31.326276 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:31.328276 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:31.328799 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:31.328921 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:31.330246 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:31.330251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:31.330255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:31.330275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:31.330332 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:31.330348 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:31.330441 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:31.330643 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:12:31.330691 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:12:31.334537 21979 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } }
I20260619 13:12:31.334982 21979 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:31.335088 21979 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4c04b72002443d8b20d94e800a96c9, State: Initialized, Role: FOLLOWER
I20260619 13:12:31.335850 21979 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } }
I20260619 13:12:31.336154 21979 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:31.336300 21979 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:31.336485 21979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:31.339161 21979 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } }
I20260619 13:12:31.339716 21979 leader_election.cc:302] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4c04b72002443d8b20d94e800a96c9; no voters:
I20260619 13:12:31.340396 21979 leader_election.cc:288] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:31.340569 21989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:31.341395 21989 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [term 1 LEADER]: Becoming Leader. State: Replica: ae4c04b72002443d8b20d94e800a96c9, State: Running, Role: LEADER
I20260619 13:12:31.342114 21989 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } }
I20260619 13:12:31.342779 21979 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:31.345808 21991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4c04b72002443d8b20d94e800a96c9. Latest consensus state: current_term: 1 leader_uuid: "ae4c04b72002443d8b20d94e800a96c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } } }
I20260619 13:12:31.345718 21990 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4c04b72002443d8b20d94e800a96c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4c04b72002443d8b20d94e800a96c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32877 } } }
I20260619 13:12:31.346100 21991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:31.346107 21990 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:31.347882 21997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:31.353799 21997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:31.364388 21997 catalog_manager.cc:1347] Generated new cluster ID: 4f81def1d2474043afc189a80495824a
I20260619 13:12:31.364524 21997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:31.385058 21997 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:31.386981 21997 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:31.397869 21997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4c04b72002443d8b20d94e800a96c9: Generated new TSK 0
I20260619 13:12:31.399056 21997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:31.414304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21912
W20260619 13:12:31.415647 21974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:55353
W20260619 13:12:31.416599 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 21912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:31.416791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 21912
W20260619 13:12:31.434458 21897 connection.cc:535] client connection to 127.0.0.1:32877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32877): Connection reset by peer (error 104)
2026-06-19T13:12:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:31 dist-test-slave-q11v krb5kdc[22010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:31 dist-test-slave-q11v krb5kdc[22010](info): set up 2 sockets
Jun 19 13:12:31 dist-test-slave-q11v krb5kdc[22010](info): commencing operation
krb5kdc: starting...
W20260619 13:12:32.554102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:32 dist-test-slave-q11v krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:12:32.821264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:33.161195 22023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:33.161484 22023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:33.161690 22023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:33.168948 22023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:33.169066 22023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:33.169170 22023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:33.169274 22023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:33.169353 22023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:33.179490 22023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:33.180835 22023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:33.182246 22023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:33.194216 22028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:33.194216 22029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:33.194273 22031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:33.195372 22023 server_base.cc:1034] running on GCE node
I20260619 13:12:33.196293 22023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:33.197777 22023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:33.198985 22023 hybrid_clock.cc:648] HybridClock initialized: now 1781874753198919 us; error 61 us; skew 500 ppm
I20260619 13:12:33.199355 22023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:33 dist-test-slave-q11v krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:33.207048 22023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:12:33.208659 22023 webserver.cc:460] Webserver started at http://127.0.0.1:42057/ using document root <none> and password file <none>
I20260619 13:12:33.209407 22023 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:33.209565 22023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:33.209901 22023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:33.213008 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "881b8be7318e45efaffc0354c8370608"
format_stamp: "Formatted at 2026-06-19 13:12:33 on dist-test-slave-q11v"
server_key: "7cbdc6942288c354075d7f169c5658e0"
server_key_iv: "c92c4588ecc43785c21523175fe2b0a4"
server_key_version: "encryptionkey@0"
I20260619 13:12:33.213855 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "881b8be7318e45efaffc0354c8370608"
format_stamp: "Formatted at 2026-06-19 13:12:33 on dist-test-slave-q11v"
server_key: "7cbdc6942288c354075d7f169c5658e0"
server_key_iv: "c92c4588ecc43785c21523175fe2b0a4"
server_key_version: "encryptionkey@0"
I20260619 13:12:33.219328 22023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:12:33.222853 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:33.224270 22023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:33.224478 22023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "881b8be7318e45efaffc0354c8370608"
format_stamp: "Formatted at 2026-06-19 13:12:33 on dist-test-slave-q11v"
server_key: "7cbdc6942288c354075d7f169c5658e0"
server_key_iv: "c92c4588ecc43785c21523175fe2b0a4"
server_key_version: "encryptionkey@0"
I20260619 13:12:33.224709 22023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:33.230645 22023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:33.234942 22023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:33.235250 22023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:33.253943 22023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46669
I20260619 13:12:33.254120 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46669 every 8 connection(s)
I20260619 13:12:33.255406 22023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:33.256575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22023
I20260619 13:12:33.256770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:33.257227 388 external_mini_cluster.cc:1428] Setting key 5697ecbe08a2e97e2d77553cb67c72ca
I20260619 13:12:33.260003 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:33.266280 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:33.259150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:45553 (local address 127.0.0.1:46669)
0619 13:12:33.259927 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:33.259944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:33.260001 (+ 57us) server_negotiation.cc:407] Connection header received
0619 13:12:33.262356 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:33.262386 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:33.262820 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:33.263303 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:33.264290 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:33.264303 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:33.264319 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:33.264337 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:33.264432 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:33.264466 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:33.264687 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:33.264818 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:12:33.265014 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":325,"threads_started":1}
I20260619 13:12:33.270192 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:33.273262 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608: Bootstrap starting.
I20260619 13:12:33.276916 22090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:33.276962 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:33.274582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:49227 (local address 127.0.0.1:46669)
0619 13:12:33.274790 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:33.274797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:33.275103 (+ 306us) server_negotiation.cc:407] Connection header received
0619 13:12:33.275235 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:33.275240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:33.275299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:33.275415 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:33.276285 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:33.276290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:33.276295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:33.276312 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:33.276354 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:33.276374 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:33.276487 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:33.276581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 13:12:33.276640 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:12:33.278223 22090 log.cc:824] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:33.281087 22090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608: No bootstrap required, opened a new log
I20260619 13:12:33.288491 22090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260619 13:12:33.288892 22090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:33.289018 22090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881b8be7318e45efaffc0354c8370608, State: Initialized, Role: FOLLOWER
I20260619 13:12:33.289727 22090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260619 13:12:33.289958 22090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:33.290112 22090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:33.290288 22090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:33.292743 22090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260619 13:12:33.293246 22090 leader_election.cc:302] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881b8be7318e45efaffc0354c8370608; no voters:
I20260619 13:12:33.293923 22090 leader_election.cc:288] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:33.294121 22100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:33.294900 22100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [term 1 LEADER]: Becoming Leader. State: Replica: 881b8be7318e45efaffc0354c8370608, State: Running, Role: LEADER
I20260619 13:12:33.295473 22100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } }
I20260619 13:12:33.296130 22090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:33.299045 22101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881b8be7318e45efaffc0354c8370608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } } }
I20260619 13:12:33.299166 22102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881b8be7318e45efaffc0354c8370608. Latest consensus state: current_term: 1 leader_uuid: "881b8be7318e45efaffc0354c8370608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881b8be7318e45efaffc0354c8370608" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46669 } } }
I20260619 13:12:33.299554 22101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:33.299553 22102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:33.300881 22108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:33.307438 22108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:33.319171 22108 catalog_manager.cc:1347] Generated new cluster ID: faaf749f13bb46688cfe4e01b7881e61
I20260619 13:12:33.319312 22108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:33.334882 22108 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:33.335495 22108 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:33.355325 22108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881b8be7318e45efaffc0354c8370608: Generated new TSK 0
I20260619 13:12:33.356271 22108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:33.370160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22023
W20260619 13:12:33.371809 22085 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:45553
W20260619 13:12:33.372568 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22023: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:33.372718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22023
2026-06-19T13:12:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:33 dist-test-slave-q11v krb5kdc[22121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:33 dist-test-slave-q11v krb5kdc[22121](info): set up 2 sockets
Jun 19 13:12:33 dist-test-slave-q11v krb5kdc[22121](info): commencing operation
krb5kdc: starting...
W20260619 13:12:34.478122 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.012s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:34 dist-test-slave-q11v krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:34.741586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:35.076509 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:35.076824 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:35.077003 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:35.084403 22134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:35.084532 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:35.084642 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:35.084765 22134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:35.084851 22134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:35.094864 22134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:35.096256 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:35.097654 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:35.109439 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:35.109499 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:35.109479 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:35.110790 22134 server_base.cc:1034] running on GCE node
I20260619 13:12:35.111608 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:35.113125 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:35.114356 22134 hybrid_clock.cc:648] HybridClock initialized: now 1781874755114324 us; error 61 us; skew 500 ppm
I20260619 13:12:35.114733 22134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:35 dist-test-slave-q11v krb5kdc[22121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:35.118062 22134 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:35.119736 22134 webserver.cc:460] Webserver started at http://127.0.97.62:34403/ using document root <none> and password file <none>
I20260619 13:12:35.120479 22134 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:35.120637 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:35.120957 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:35.124085 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "566f383ca8344e71a13a3d3b7fae362e"
format_stamp: "Formatted at 2026-06-19 13:12:35 on dist-test-slave-q11v"
server_key: "34f0a29975595d653aa1224a773c004d"
server_key_iv: "e1703dc698e61be5314a0a621f36a87d"
server_key_version: "encryptionkey@0"
I20260619 13:12:35.124866 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "566f383ca8344e71a13a3d3b7fae362e"
format_stamp: "Formatted at 2026-06-19 13:12:35 on dist-test-slave-q11v"
server_key: "34f0a29975595d653aa1224a773c004d"
server_key_iv: "e1703dc698e61be5314a0a621f36a87d"
server_key_version: "encryptionkey@0"
I20260619 13:12:35.131454 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260619 13:12:35.134979 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:35.136323 22134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:35.136508 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "566f383ca8344e71a13a3d3b7fae362e"
format_stamp: "Formatted at 2026-06-19 13:12:35 on dist-test-slave-q11v"
server_key: "34f0a29975595d653aa1224a773c004d"
server_key_iv: "e1703dc698e61be5314a0a621f36a87d"
server_key_version: "encryptionkey@0"
I20260619 13:12:35.136711 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:35.162818 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:35.167026 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:35.167346 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:35.186374 22134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32853
I20260619 13:12:35.186415 22200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32853 every 8 connection(s)
I20260619 13:12:35.187937 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:35.191843 22201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:35.197767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22134
I20260619 13:12:35.197968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:35.198407 388 external_mini_cluster.cc:1428] Setting key 1eda88b35f73774f108b08605d162a67
I20260619 13:12:35.202973 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e: Bootstrap starting.
I20260619 13:12:35.206784 22201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:35.208124 22201 log.cc:824] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:35.211071 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e: No bootstrap required, opened a new log
Jun 19 13:12:35 dist-test-slave-q11v krb5kdc[22121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:35.219063 22201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } }
I20260619 13:12:35.219475 22201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:35.219625 22201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566f383ca8344e71a13a3d3b7fae362e, State: Initialized, Role: FOLLOWER
I20260619 13:12:35.220304 22201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } }
I20260619 13:12:35.220566 22201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:35.220716 22201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:35.220914 22201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:35.222440 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:35.200360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54445 (local address 127.0.97.62:32853)
0619 13:12:35.201212 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:35.201223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:35.201263 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:12:35.203024 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:35.203076 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:35.203510 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:35.203841 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:35.205704 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:35.207066 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:35.209042 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:35.209865 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:35.210054 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:35.214537 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:35.214548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:35.214564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:35.214604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:35.217924 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:35.218789 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:35.218796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:35.218803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:35.218960 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:35.219441 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:35.219448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:35.219456 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:35.220100 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:35.220449 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:35.220925 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 13:12:35.221285 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":416,"threads_started":1}
I20260619 13:12:35.223553 22201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } }
I20260619 13:12:35.224191 22201 leader_election.cc:302] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 566f383ca8344e71a13a3d3b7fae362e; no voters:
I20260619 13:12:35.224845 22201 leader_election.cc:288] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:35.225008 22206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:35.226524 22206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [term 1 LEADER]: Becoming Leader. State: Replica: 566f383ca8344e71a13a3d3b7fae362e, State: Running, Role: LEADER
I20260619 13:12:35.227058 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:35.227025 22201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:35.227206 22206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } }
I20260619 13:12:35.236503 22207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "566f383ca8344e71a13a3d3b7fae362e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } } }
I20260619 13:12:35.236966 22207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:35.237407 22208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 566f383ca8344e71a13a3d3b7fae362e. Latest consensus state: current_term: 1 leader_uuid: "566f383ca8344e71a13a3d3b7fae362e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566f383ca8344e71a13a3d3b7fae362e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32853 } } }
I20260619 13:12:35.237723 22208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:35.239657 22219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:35.244998 22219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:35.264569 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:35.235075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51143 (local address 127.0.97.62:32853)
0619 13:12:35.235233 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:35.235241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:35.238792 (+ 3551us) server_negotiation.cc:407] Connection header received
0619 13:12:35.241640 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:35.241653 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:35.241786 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:35.241936 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:35.244591 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:35.246279 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:35.248813 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:35.249698 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:35.249797 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:35.254099 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:35.254110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:35.254119 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:35.254190 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:35.259435 (+ 5245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:35.260885 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:35.260895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:35.260901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:35.261102 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:35.261884 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:35.261892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:35.261897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:35.262452 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:35.262764 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:35.263706 (+ 942us) server_negotiation.cc:299] Negotiation successful
0619 13:12:35.263987 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":34}
I20260619 13:12:35.268796 22219 catalog_manager.cc:1347] Generated new cluster ID: 3f8fcfc3bdf443de8051adcf1d1ddee3
I20260619 13:12:35.268920 22219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:35.292717 22219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:35.294174 22219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:35.305024 22219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 566f383ca8344e71a13a3d3b7fae362e: Generated new TSK 0
I20260619 13:12:35.306020 22219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:35.323872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22134
I20260619 13:12:35.326108 22196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:35.421674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22134
W20260619 13:12:35.439450 22119 connection.cc:535] client connection to 127.0.97.62:32853 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:32853): Connection reset by peer (error 104)
2026-06-19T13:12:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:35 dist-test-slave-q11v krb5kdc[22233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:35 dist-test-slave-q11v krb5kdc[22233](info): set up 2 sockets
Jun 19 13:12:35 dist-test-slave-q11v krb5kdc[22233](info): commencing operation
krb5kdc: starting...
W20260619 13:12:36.530102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:36 dist-test-slave-q11v krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:36.788563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:37.122179 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:37.122499 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:37.122666 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:37.130179 22246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:37.130307 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:37.130388 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:37.130496 22246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:37.130574 22246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:37.142031 22246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:37.143429 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:37.144775 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:37.156740 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:37.156692 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:37.156780 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:37.157660 22246 server_base.cc:1034] running on GCE node
I20260619 13:12:37.158555 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:37.160102 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:37.161343 22246 hybrid_clock.cc:648] HybridClock initialized: now 1781874757161291 us; error 45 us; skew 500 ppm
I20260619 13:12:37.161763 22246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:37 dist-test-slave-q11v krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:37.164850 22246 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:37.166451 22246 webserver.cc:460] Webserver started at http://127.0.97.62:41333/ using document root <none> and password file <none>
I20260619 13:12:37.167109 22246 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:37.167217 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:37.167467 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:37.170495 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "d8051097bae04077a613a492ddba8cf5"
format_stamp: "Formatted at 2026-06-19 13:12:37 on dist-test-slave-q11v"
server_key: "aa91779a6366f34f766772058d980870"
server_key_iv: "38d6b46edc235a2c36f1b4eaeaf1ffca"
server_key_version: "encryptionkey@0"
I20260619 13:12:37.171212 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "d8051097bae04077a613a492ddba8cf5"
format_stamp: "Formatted at 2026-06-19 13:12:37 on dist-test-slave-q11v"
server_key: "aa91779a6366f34f766772058d980870"
server_key_iv: "38d6b46edc235a2c36f1b4eaeaf1ffca"
server_key_version: "encryptionkey@0"
I20260619 13:12:37.176411 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:12:37.179972 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:37.181389 22246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:12:37.181573 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "d8051097bae04077a613a492ddba8cf5"
format_stamp: "Formatted at 2026-06-19 13:12:37 on dist-test-slave-q11v"
server_key: "aa91779a6366f34f766772058d980870"
server_key_iv: "38d6b46edc235a2c36f1b4eaeaf1ffca"
server_key_version: "encryptionkey@0"
I20260619 13:12:37.181844 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:37.196712 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:37.201077 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:37.201383 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:37.220578 22246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40109
I20260619 13:12:37.220614 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40109 every 8 connection(s)
I20260619 13:12:37.222092 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:37.223694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22246
I20260619 13:12:37.223872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:37.224440 388 external_mini_cluster.cc:1428] Setting key 80bb5db0494cd9655c4d582fa7b2225a
I20260619 13:12:37.226954 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:37.238232 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:37.226391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34549 (local address 127.0.97.62:40109)
0619 13:12:37.227230 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:37.227246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:37.227290 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:12:37.228952 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:37.228974 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:37.229383 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:37.229861 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:37.230441 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:37.231775 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:37.233715 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:37.234362 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:37.234493 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:37.235834 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:37.235842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:37.235859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:37.236410 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:37.236486 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:37.236510 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:37.236640 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:37.236913 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:12:37.237143 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260619 13:12:37.239701 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5: Bootstrap starting.
I20260619 13:12:37.242276 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:37.243140 22313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:37.244117 22313 log.cc:824] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:37.247126 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5: No bootstrap required, opened a new log
I20260619 13:12:37.253772 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:37.246676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57233 (local address 127.0.97.62:40109)
0619 13:12:37.246859 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:37.246865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:37.247309 (+ 444us) server_negotiation.cc:407] Connection header received
0619 13:12:37.247495 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:37.247500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:37.247581 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:37.247718 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:37.248148 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:37.249292 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:37.250775 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:37.251485 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:37.251608 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:37.252822 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:37.252828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:37.252833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:37.252856 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:37.252908 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:37.252930 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:37.253102 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:37.253374 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 13:12:37.253428 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:12:37.256496 22313 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } }
I20260619 13:12:37.256855 22313 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:37.256928 22313 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8051097bae04077a613a492ddba8cf5, State: Initialized, Role: FOLLOWER
I20260619 13:12:37.257426 22313 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } }
I20260619 13:12:37.257704 22313 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:37.257817 22313 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:37.257943 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:37.260100 22313 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } }
I20260619 13:12:37.260598 22313 leader_election.cc:302] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8051097bae04077a613a492ddba8cf5; no voters:
I20260619 13:12:37.261243 22313 leader_election.cc:288] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:37.261379 22323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:37.262221 22323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [term 1 LEADER]: Becoming Leader. State: Replica: d8051097bae04077a613a492ddba8cf5, State: Running, Role: LEADER
I20260619 13:12:37.262814 22323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } }
I20260619 13:12:37.263443 22313 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:37.265466 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8051097bae04077a613a492ddba8cf5. Latest consensus state: current_term: 1 leader_uuid: "d8051097bae04077a613a492ddba8cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } } }
I20260619 13:12:37.265990 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:37.266165 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8051097bae04077a613a492ddba8cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8051097bae04077a613a492ddba8cf5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40109 } } }
I20260619 13:12:37.266525 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:37.270828 22331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:37.275089 22331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:37.283274 22331 catalog_manager.cc:1347] Generated new cluster ID: a731e6939d8b4856a0abb080487303c4
I20260619 13:12:37.283407 22331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:37.292708 22331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:37.294682 22331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:37.318959 22331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8051097bae04077a613a492ddba8cf5: Generated new TSK 0
I20260619 13:12:37.320163 22331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:37.330211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22246
W20260619 13:12:37.331425 22308 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34549
W20260619 13:12:37.332222 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:37.332365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22246
W20260619 13:12:37.346475 22231 connection.cc:535] client connection to 127.0.97.62:40109 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40109): Connection reset by peer (error 104)
2026-06-19T13:12:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:37 dist-test-slave-q11v krb5kdc[22344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:37 dist-test-slave-q11v krb5kdc[22344](info): set up 2 sockets
Jun 19 13:12:37 dist-test-slave-q11v krb5kdc[22344](info): commencing operation
krb5kdc: starting...
W20260619 13:12:38.462284 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:38 dist-test-slave-q11v krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:38.734495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34185
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:39.070099 22357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:39.070400 22357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:39.070591 22357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:39.077855 22357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:39.077967 22357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:39.078092 22357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:39.078197 22357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:39.078262 22357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:39.088366 22357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:39.089797 22357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:39.091190 22357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:39.103698 22365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:39.103660 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:39.103682 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:39.104650 22357 server_base.cc:1034] running on GCE node
I20260619 13:12:39.105475 22357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:39.107012 22357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:39.108222 22357 hybrid_clock.cc:648] HybridClock initialized: now 1781874759108182 us; error 37 us; skew 500 ppm
I20260619 13:12:39.108590 22357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:39 dist-test-slave-q11v krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:39.117725 22357 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:39.119478 22357 webserver.cc:460] Webserver started at http://127.0.97.62:34711/ using document root <none> and password file <none>
I20260619 13:12:39.120241 22357 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:39.120394 22357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:39.120718 22357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:39.123867 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "b8481e96b45d427d835c301d0fec9403"
format_stamp: "Formatted at 2026-06-19 13:12:39 on dist-test-slave-q11v"
server_key: "e30417f2afc9bd1b89d4f111a8ad12e2"
server_key_iv: "551e3bbd79dab243bd5982b9d649b324"
server_key_version: "encryptionkey@0"
I20260619 13:12:39.124715 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8481e96b45d427d835c301d0fec9403"
format_stamp: "Formatted at 2026-06-19 13:12:39 on dist-test-slave-q11v"
server_key: "e30417f2afc9bd1b89d4f111a8ad12e2"
server_key_iv: "551e3bbd79dab243bd5982b9d649b324"
server_key_version: "encryptionkey@0"
I20260619 13:12:39.130280 22357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:12:39.133853 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:39.135246 22357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:39.135440 22357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "b8481e96b45d427d835c301d0fec9403"
format_stamp: "Formatted at 2026-06-19 13:12:39 on dist-test-slave-q11v"
server_key: "e30417f2afc9bd1b89d4f111a8ad12e2"
server_key_iv: "551e3bbd79dab243bd5982b9d649b324"
server_key_version: "encryptionkey@0"
I20260619 13:12:39.135658 22357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:39.142073 22357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:39.147174 22357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:39.147500 22357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:39.166355 22357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34185
I20260619 13:12:39.166391 22423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34185 every 8 connection(s)
I20260619 13:12:39.167877 22357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:39.170123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22357
I20260619 13:12:39.170323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:39.170790 388 external_mini_cluster.cc:1428] Setting key c92e3dd885e39731a3fedb3b828738c8
I20260619 13:12:39.172761 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:39.181000 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:39.172798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55801 (local address 127.0.97.62:34185)
0619 13:12:39.173733 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:39.173749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:39.173811 (+ 62us) server_negotiation.cc:407] Connection header received
0619 13:12:39.176230 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:39.176263 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:39.176809 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:39.177370 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:39.178662 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:39.178672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:39.178704 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:39.178725 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:39.178815 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:39.178854 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:39.179114 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:39.179257 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 13:12:39.179518 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":355,"threads_started":1}
I20260619 13:12:39.185225 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:39.187904 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403: Bootstrap starting.
I20260619 13:12:39.191727 22424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:39.192401 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:39.189681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53981 (local address 127.0.97.62:34185)
0619 13:12:39.189847 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:39.189852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:39.190347 (+ 495us) server_negotiation.cc:407] Connection header received
0619 13:12:39.190558 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:39.190563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:39.190627 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:39.190734 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:39.191696 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:39.191707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:39.191716 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:39.191734 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:39.191782 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:39.191818 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:39.191931 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:39.192042 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 13:12:39.192117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:12:39.193032 22424 log.cc:824] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:39.195935 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403: No bootstrap required, opened a new log
I20260619 13:12:39.203482 22424 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } }
I20260619 13:12:39.203940 22424 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:39.204062 22424 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8481e96b45d427d835c301d0fec9403, State: Initialized, Role: FOLLOWER
I20260619 13:12:39.204671 22424 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } }
I20260619 13:12:39.204927 22424 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:39.205090 22424 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:39.205269 22424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:39.207638 22424 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } }
I20260619 13:12:39.208221 22424 leader_election.cc:302] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8481e96b45d427d835c301d0fec9403; no voters:
I20260619 13:12:39.208846 22424 leader_election.cc:288] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:39.209061 22434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:39.209863 22434 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [term 1 LEADER]: Becoming Leader. State: Replica: b8481e96b45d427d835c301d0fec9403, State: Running, Role: LEADER
I20260619 13:12:39.210556 22434 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } }
I20260619 13:12:39.211180 22424 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:39.213876 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8481e96b45d427d835c301d0fec9403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } } }
I20260619 13:12:39.213976 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8481e96b45d427d835c301d0fec9403. Latest consensus state: current_term: 1 leader_uuid: "b8481e96b45d427d835c301d0fec9403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8481e96b45d427d835c301d0fec9403" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34185 } } }
I20260619 13:12:39.214300 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:39.214380 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:39.218096 22441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:39.223726 22441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:39.237391 22441 catalog_manager.cc:1347] Generated new cluster ID: 7aa2c7b71736428a9288c25b45edde48
I20260619 13:12:39.237520 22441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:39.263638 22441 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:39.264246 22441 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:39.281845 22441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8481e96b45d427d835c301d0fec9403: Generated new TSK 0
I20260619 13:12:39.283170 22441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:39.288375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22357
W20260619 13:12:39.289826 22419 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55801
W20260619 13:12:39.290645 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:39.290802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22357
2026-06-19T13:12:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:39 dist-test-slave-q11v krb5kdc[22455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:39 dist-test-slave-q11v krb5kdc[22455](info): set up 2 sockets
Jun 19 13:12:39 dist-test-slave-q11v krb5kdc[22455](info): commencing operation
krb5kdc: starting...
W20260619 13:12:40.402123 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:40 dist-test-slave-q11v krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:40.667425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33531
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:41.001559 22468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:41.001950 22468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:41.002133 22468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:41.009843 22468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:41.009960 22468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:41.010098 22468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:41.010205 22468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:41.010289 22468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:41.020836 22468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:41.022310 22468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:41.023648 22468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:41.035588 22476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:41.035640 22474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:41.035555 22473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:41.037704 22468 server_base.cc:1034] running on GCE node
I20260619 13:12:41.038594 22468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:41.040135 22468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:41.041397 22468 hybrid_clock.cc:648] HybridClock initialized: now 1781874761041350 us; error 52 us; skew 500 ppm
I20260619 13:12:41.041855 22468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:41 dist-test-slave-q11v krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:41.045313 22468 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:41.046854 22468 webserver.cc:460] Webserver started at http://127.0.97.62:43941/ using document root <none> and password file <none>
I20260619 13:12:41.047616 22468 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:41.047765 22468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:41.048086 22468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:41.051205 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "f18c8fa0beff4ea393bf6a7e12af2166"
format_stamp: "Formatted at 2026-06-19 13:12:41 on dist-test-slave-q11v"
server_key: "6f19a956db8242a3244ed6b346f92395"
server_key_iv: "6141f9cbf4e8428244a0cbaada172e03"
server_key_version: "encryptionkey@0"
I20260619 13:12:41.052028 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "f18c8fa0beff4ea393bf6a7e12af2166"
format_stamp: "Formatted at 2026-06-19 13:12:41 on dist-test-slave-q11v"
server_key: "6f19a956db8242a3244ed6b346f92395"
server_key_iv: "6141f9cbf4e8428244a0cbaada172e03"
server_key_version: "encryptionkey@0"
I20260619 13:12:41.057459 22468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:12:41.061197 22483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:41.062718 22468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:41.062927 22468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "f18c8fa0beff4ea393bf6a7e12af2166"
format_stamp: "Formatted at 2026-06-19 13:12:41 on dist-test-slave-q11v"
server_key: "6f19a956db8242a3244ed6b346f92395"
server_key_iv: "6141f9cbf4e8428244a0cbaada172e03"
server_key_version: "encryptionkey@0"
I20260619 13:12:41.063171 22468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:41.083578 22468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:41.088164 22468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:41.088469 22468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:41.107738 22468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33531
I20260619 13:12:41.107767 22534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33531 every 8 connection(s)
I20260619 13:12:41.109213 22468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:41.112967 22535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:41.113176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22468
I20260619 13:12:41.113368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:41.113899 388 external_mini_cluster.cc:1428] Setting key 4533837cf1a868890e64fc996cd309bf
I20260619 13:12:41.127964 22535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166: Bootstrap starting.
Jun 19 13:12:41 dist-test-slave-q11v krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:41.131779 22535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:41.133232 22535 log.cc:824] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:41.136682 22535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166: No bootstrap required, opened a new log
I20260619 13:12:41.139377 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:41.115734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33861 (local address 127.0.97.62:33531)
0619 13:12:41.116616 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:41.116633 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:41.116687 (+ 54us) server_negotiation.cc:407] Connection header received
0619 13:12:41.118486 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:41.118520 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:41.119103 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:41.119583 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:41.120852 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:41.122839 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:41.124288 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:41.125454 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:41.125679 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:41.130275 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:41.130284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:41.130302 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:41.130359 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:41.134443 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:41.135177 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:41.135185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:41.135194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:41.135400 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:41.135793 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:41.135799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:41.135802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:41.136513 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:41.136809 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:41.137399 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 13:12:41.137878 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":360,"threads_started":1}
I20260619 13:12:41.143469 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:41.146211 22535 raft_consensus.cc:348] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } }
I20260619 13:12:41.146698 22535 raft_consensus.cc:374] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:41.146826 22535 raft_consensus.cc:729] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18c8fa0beff4ea393bf6a7e12af2166, State: Initialized, Role: FOLLOWER
I20260619 13:12:41.147480 22535 consensus_queue.cc:260] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } }
I20260619 13:12:41.147727 22535 raft_consensus.cc:388] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:41.147895 22535 raft_consensus.cc:482] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:41.148116 22535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:41.151367 22535 raft_consensus.cc:504] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } }
I20260619 13:12:41.152016 22535 leader_election.cc:302] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18c8fa0beff4ea393bf6a7e12af2166; no voters:
I20260619 13:12:41.152684 22535 leader_election.cc:288] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:41.153692 22545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:41.154400 22545 raft_consensus.cc:686] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [term 1 LEADER]: Becoming Leader. State: Replica: f18c8fa0beff4ea393bf6a7e12af2166, State: Running, Role: LEADER
I20260619 13:12:41.154987 22535 sys_catalog.cc:564] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:41.155052 22545 consensus_queue.cc:237] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } }
I20260619 13:12:41.158423 22547 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f18c8fa0beff4ea393bf6a7e12af2166. Latest consensus state: current_term: 1 leader_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } } }
I20260619 13:12:41.158310 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c8fa0beff4ea393bf6a7e12af2166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33531 } } }
I20260619 13:12:41.158840 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:41.158840 22547 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:41.165990 22552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:41.169898 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:41.148026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43399 (local address 127.0.97.62:33531)
0619 13:12:41.148251 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:41.148261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:41.150258 (+ 1997us) server_negotiation.cc:407] Connection header received
0619 13:12:41.150465 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:41.150471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:41.150565 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:41.150659 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:41.152650 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:41.153886 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:41.155925 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:41.156842 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:41.157030 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:41.161133 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:41.161143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:41.161151 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:41.161208 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:41.165626 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:41.166625 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:41.166630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:41.166634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:41.166776 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:41.167320 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:41.167324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:41.167327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:41.167742 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:41.168015 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:41.169233 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0619 13:12:41.169473 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:12:41.171692 22552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:41.180902 22552 catalog_manager.cc:1347] Generated new cluster ID: ea9ac8108aa14123bcbd2c0aeddfa963
I20260619 13:12:41.181025 22552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:41.206565 22552 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:41.208014 22552 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:41.218616 22552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f18c8fa0beff4ea393bf6a7e12af2166: Generated new TSK 0
I20260619 13:12:41.219529 22552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:41.227007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22468
I20260619 13:12:41.228191 22530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:41.320804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22468
2026-06-19T13:12:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:41 dist-test-slave-q11v krb5kdc[22567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:41 dist-test-slave-q11v krb5kdc[22567](info): set up 2 sockets
Jun 19 13:12:41 dist-test-slave-q11v krb5kdc[22567](info): commencing operation
krb5kdc: starting...
W20260619 13:12:42.454159 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:42 dist-test-slave-q11v krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:42.718178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42127
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:43.057212 22580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:43.057516 22580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:43.057703 22580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:43.065176 22580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:43.065304 22580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:43.065353 22580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:43.065423 22580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:43.065467 22580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:43.075659 22580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:43.077009 22580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:43.078347 22580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:43.090653 22585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:43.090552 22586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:43.090602 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:43.092734 22580 server_base.cc:1034] running on GCE node
I20260619 13:12:43.094448 22580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:43.096141 22580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:43.097401 22580 hybrid_clock.cc:648] HybridClock initialized: now 1781874763097361 us; error 44 us; skew 500 ppm
I20260619 13:12:43.097852 22580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:43 dist-test-slave-q11v krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:43.101086 22580 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:43.102828 22580 webserver.cc:460] Webserver started at http://127.0.97.62:33839/ using document root <none> and password file <none>
I20260619 13:12:43.103564 22580 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:43.103715 22580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:43.104044 22580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:43.107257 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "ad44174855e146a3bce013d93573af25"
format_stamp: "Formatted at 2026-06-19 13:12:43 on dist-test-slave-q11v"
server_key: "232e05c953aebdb113ffee9640d89c21"
server_key_iv: "feee6f5341f1c580f091c5ccf83183e6"
server_key_version: "encryptionkey@0"
I20260619 13:12:43.108080 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad44174855e146a3bce013d93573af25"
format_stamp: "Formatted at 2026-06-19 13:12:43 on dist-test-slave-q11v"
server_key: "232e05c953aebdb113ffee9640d89c21"
server_key_iv: "feee6f5341f1c580f091c5ccf83183e6"
server_key_version: "encryptionkey@0"
I20260619 13:12:43.113547 22580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 13:12:43.117215 22595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:43.118741 22580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:43.118934 22580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "ad44174855e146a3bce013d93573af25"
format_stamp: "Formatted at 2026-06-19 13:12:43 on dist-test-slave-q11v"
server_key: "232e05c953aebdb113ffee9640d89c21"
server_key_iv: "feee6f5341f1c580f091c5ccf83183e6"
server_key_version: "encryptionkey@0"
I20260619 13:12:43.119148 22580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:43.136654 22580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:43.141045 22580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:43.141407 22580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:43.162343 22580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42127
I20260619 13:12:43.162365 22646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42127 every 8 connection(s)
I20260619 13:12:43.163776 22580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:43.164310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22580
I20260619 13:12:43.164515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:43.165154 388 external_mini_cluster.cc:1428] Setting key 09042fe37984979b39d5c4bc6af2b60b
I20260619 13:12:43.168761 22647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:43.178416 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:43.167122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35213 (local address 127.0.97.62:42127)
0619 13:12:43.167933 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:43.167946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:43.167990 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:12:43.169744 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:43.169768 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:43.170164 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:43.170570 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:43.171113 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:43.172450 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:43.173856 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:43.174553 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:43.174766 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:43.175996 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:43.176004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:43.176018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:43.176566 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:43.176653 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:43.176677 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:43.176824 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:43.177045 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 13:12:43.177318 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":367,"threads_started":1}
I20260619 13:12:43.181202 22647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25: Bootstrap starting.
I20260619 13:12:43.182492 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:43.185459 22647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:43.187136 22647 log.cc:824] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:43.190989 22647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25: No bootstrap required, opened a new log
I20260619 13:12:43.195502 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:43.188235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52415 (local address 127.0.97.62:42127)
0619 13:12:43.188476 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:43.188484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:43.188986 (+ 502us) server_negotiation.cc:407] Connection header received
0619 13:12:43.189185 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:43.189189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:43.189249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:43.189333 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:43.190036 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:43.190745 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:43.192852 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:43.193329 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:43.193427 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:43.194734 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:43.194739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:43.194745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:43.194765 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:43.194804 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:43.194820 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:43.194896 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:43.195069 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 13:12:43.195152 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:12:43.200812 22647 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } }
I20260619 13:12:43.201257 22647 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:43.201395 22647 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad44174855e146a3bce013d93573af25, State: Initialized, Role: FOLLOWER
I20260619 13:12:43.202209 22647 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } }
I20260619 13:12:43.202527 22647 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:43.202693 22647 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:43.202903 22647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:43.205309 22647 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } }
I20260619 13:12:43.206001 22647 leader_election.cc:302] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad44174855e146a3bce013d93573af25; no voters:
I20260619 13:12:43.206677 22647 leader_election.cc:288] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:43.206795 22657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:43.207686 22657 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [term 1 LEADER]: Becoming Leader. State: Replica: ad44174855e146a3bce013d93573af25, State: Running, Role: LEADER
I20260619 13:12:43.208395 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } }
I20260619 13:12:43.209262 22647 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:43.211740 22658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad44174855e146a3bce013d93573af25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } } }
I20260619 13:12:43.212327 22658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:43.212285 22659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad44174855e146a3bce013d93573af25. Latest consensus state: current_term: 1 leader_uuid: "ad44174855e146a3bce013d93573af25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44174855e146a3bce013d93573af25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42127 } } }
I20260619 13:12:43.212685 22659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:43.218729 22666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:43.224205 22666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:43.233778 22666 catalog_manager.cc:1347] Generated new cluster ID: 6aec4bfd3f3e4fee8bb7b3e74ba71c4a
I20260619 13:12:43.233948 22666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:43.242116 22666 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:43.244019 22666 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:43.259557 22666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad44174855e146a3bce013d93573af25: Generated new TSK 0
I20260619 13:12:43.260748 22666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:43.266532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22580
W20260619 13:12:43.267810 22642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35213
W20260619 13:12:43.268530 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:43.268683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22580
2026-06-19T13:12:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:43 dist-test-slave-q11v krb5kdc[22678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:43 dist-test-slave-q11v krb5kdc[22678](info): set up 2 sockets
Jun 19 13:12:43 dist-test-slave-q11v krb5kdc[22678](info): commencing operation
krb5kdc: starting...
W20260619 13:12:44.382157 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:44 dist-test-slave-q11v krb5kdc[22678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:44.649212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35571
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:44.994743 22691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:44.995146 22691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:44.995340 22691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:45.002847 22691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:45.003048 22691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:45.003168 22691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:45.003288 22691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:45.003382 22691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:45.013728 22691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:45.015206 22691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:45.016613 22691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:45.029381 22696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:45.029381 22697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:45.029378 22699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:45.030535 22691 server_base.cc:1034] running on GCE node
I20260619 13:12:45.031405 22691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:45.032917 22691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:45.034153 22691 hybrid_clock.cc:648] HybridClock initialized: now 1781874765034082 us; error 47 us; skew 500 ppm
I20260619 13:12:45.034587 22691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:45 dist-test-slave-q11v krb5kdc[22678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:45.043236 22691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:45.044966 22691 webserver.cc:460] Webserver started at http://127.0.97.62:45081/ using document root <none> and password file <none>
I20260619 13:12:45.045764 22691 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:45.045914 22691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:45.046240 22691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:45.049285 22691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "9bf6809f1a9147b29b8274a805e8a38d"
format_stamp: "Formatted at 2026-06-19 13:12:45 on dist-test-slave-q11v"
server_key: "f354132420abbf9577c1a08a9a24c5c5"
server_key_iv: "071537fc4de3f41f1d27c92087ff8f12"
server_key_version: "encryptionkey@0"
I20260619 13:12:45.050091 22691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "9bf6809f1a9147b29b8274a805e8a38d"
format_stamp: "Formatted at 2026-06-19 13:12:45 on dist-test-slave-q11v"
server_key: "f354132420abbf9577c1a08a9a24c5c5"
server_key_iv: "071537fc4de3f41f1d27c92087ff8f12"
server_key_version: "encryptionkey@0"
I20260619 13:12:45.055368 22691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:12:45.058905 22706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:45.060263 22691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:45.060454 22691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "9bf6809f1a9147b29b8274a805e8a38d"
format_stamp: "Formatted at 2026-06-19 13:12:45 on dist-test-slave-q11v"
server_key: "f354132420abbf9577c1a08a9a24c5c5"
server_key_iv: "071537fc4de3f41f1d27c92087ff8f12"
server_key_version: "encryptionkey@0"
I20260619 13:12:45.060668 22691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:45.066565 22691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:45.070658 22691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:45.070979 22691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:45.090001 22691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35571
I20260619 13:12:45.090044 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35571 every 8 connection(s)
I20260619 13:12:45.091502 22691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:45.094951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22691
I20260619 13:12:45.095137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:45.095641 22758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:45.095700 388 external_mini_cluster.cc:1428] Setting key d97e390e0a8195bf5deb8aa0b00eefef
I20260619 13:12:45.104002 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:45.097792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45993 (local address 127.0.97.62:35571)
0619 13:12:45.098715 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:45.098727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:45.098774 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:12:45.100448 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:45.100472 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:45.100854 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:45.101200 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:45.102323 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:45.102329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:45.102342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:45.102359 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:45.102424 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:45.102447 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:45.102648 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:45.102752 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 13:12:45.103010 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":418,"threads_started":1}
I20260619 13:12:45.107856 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:45.109383 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d: Bootstrap starting.
I20260619 13:12:45.113099 22758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:45.114343 22758 log.cc:824] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:45.114959 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:45.112208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47331 (local address 127.0.97.62:35571)
0619 13:12:45.112371 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:45.112376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:45.112870 (+ 494us) server_negotiation.cc:407] Connection header received
0619 13:12:45.113081 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:45.113090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:45.113157 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:45.113278 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:45.114246 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:45.114252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:45.114257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:12:45.114275 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:45.114325 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:12:45.114345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:45.114472 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:45.114569 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 13:12:45.114634 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:12:45.117547 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d: No bootstrap required, opened a new log
I20260619 13:12:45.125422 22758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } }
I20260619 13:12:45.125939 22758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:12:45.126065 22758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6809f1a9147b29b8274a805e8a38d, State: Initialized, Role: FOLLOWER
I20260619 13:12:45.126737 22758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } }
I20260619 13:12:45.126993 22758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:12:45.127152 22758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:12:45.127332 22758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:45.130000 22758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } }
I20260619 13:12:45.130573 22758 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf6809f1a9147b29b8274a805e8a38d; no voters:
I20260619 13:12:45.131207 22758 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:12:45.131346 22768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:45.132037 22768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [term 1 LEADER]: Becoming Leader. State: Replica: 9bf6809f1a9147b29b8274a805e8a38d, State: Running, Role: LEADER
I20260619 13:12:45.132622 22768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } }
I20260619 13:12:45.133397 22758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:45.135862 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf6809f1a9147b29b8274a805e8a38d. Latest consensus state: current_term: 1 leader_uuid: "9bf6809f1a9147b29b8274a805e8a38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } } }
I20260619 13:12:45.135826 22769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bf6809f1a9147b29b8274a805e8a38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6809f1a9147b29b8274a805e8a38d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35571 } } }
I20260619 13:12:45.136358 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:45.136369 22769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:45.138674 22774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:45.143841 22774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:45.154179 22774 catalog_manager.cc:1347] Generated new cluster ID: 6869cf9dcc3743a097dd75a7bb9a403f
I20260619 13:12:45.154328 22774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:45.168803 22774 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:45.169281 22774 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:45.185081 22774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bf6809f1a9147b29b8274a805e8a38d: Generated new TSK 0
I20260619 13:12:45.186275 22774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:45.196925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22691
W20260619 13:12:45.198369 22753 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45993
W20260619 13:12:45.199092 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:12:45.199242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22691
2026-06-19T13:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1929 ms)
[----------] 18 tests from AuthTokenIssuingTest (35455 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:45 dist-test-slave-q11v krb5kdc[22789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:45 dist-test-slave-q11v krb5kdc[22789](info): set up 2 sockets
Jun 19 13:12:45 dist-test-slave-q11v krb5kdc[22789](info): commencing operation
krb5kdc: starting...
W20260619 13:12:46.306263 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:46 dist-test-slave-q11v krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:46.571081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39907
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:46.904142 22802 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:46.904572 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:46.904704 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:46.904851 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:46.912289 22802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:46.912446 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:46.912556 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:46.912653 22802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:46.912744 22802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:46.923538 22802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:46.924999 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:46.926438 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:46.938768 22807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:46.938839 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:46.938874 22808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:46.939735 22802 server_base.cc:1034] running on GCE node
I20260619 13:12:46.940587 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:46.942093 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:46.943310 22802 hybrid_clock.cc:648] HybridClock initialized: now 1781874766943225 us; error 54 us; skew 500 ppm
I20260619 13:12:46.943692 22802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:46 dist-test-slave-q11v krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:46.947013 22802 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:46.948467 22802 webserver.cc:460] Webserver started at http://127.0.97.62:36533/ using document root <none> and password file <none>
I20260619 13:12:46.949175 22802 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:46.949291 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:46.949669 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:46.952643 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "697a0f8e71c44c87a44416686affd9ba"
format_stamp: "Formatted at 2026-06-19 13:12:46 on dist-test-slave-q11v"
server_key: "a884e4f2442fefdd427be9e190ef21dc"
server_key_iv: "a1b45ec2e340a631dcdcae833a48011b"
server_key_version: "encryptionkey@0"
I20260619 13:12:46.953444 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "697a0f8e71c44c87a44416686affd9ba"
format_stamp: "Formatted at 2026-06-19 13:12:46 on dist-test-slave-q11v"
server_key: "a884e4f2442fefdd427be9e190ef21dc"
server_key_iv: "a1b45ec2e340a631dcdcae833a48011b"
server_key_version: "encryptionkey@0"
I20260619 13:12:46.958684 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:12:46.962203 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:46.963611 22802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:46.963811 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "697a0f8e71c44c87a44416686affd9ba"
format_stamp: "Formatted at 2026-06-19 13:12:46 on dist-test-slave-q11v"
server_key: "a884e4f2442fefdd427be9e190ef21dc"
server_key_iv: "a1b45ec2e340a631dcdcae833a48011b"
server_key_version: "encryptionkey@0"
I20260619 13:12:46.964061 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:46.983474 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:46.988090 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:46.988392 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:47.007417 22802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39907
I20260619 13:12:47.007426 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39907 every 8 connection(s)
I20260619 13:12:47.008893 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:47.012632 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:47.016597 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22802
I20260619 13:12:47.016826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:47.017472 388 external_mini_cluster.cc:1428] Setting key 82aeced86e05c5f76851c3cbbac50bf6
I20260619 13:12:47.019275 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 13:12:47.046484 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.022540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.0.1:55470)
0619 13:12:47.023409 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.023460 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:47.026143 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:47.027226 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:47.027232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:47.027896 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:47.028286 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.028306 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.030670 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.030674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.031957 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.031969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.033355 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.033364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.033687 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.034455 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:47.034478 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:47.037118 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:47.042048 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.042060 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.042071 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.042525 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.043030 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.043033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.043035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.043197 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.044244 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:47.044250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:47.044460 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 13:12:47.044689 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 13:12:47.045282 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":183,"thread_start_us":339,"threads_started":1}
I20260619 13:12:47.046505 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.023623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.97.62:39907)
0619 13:12:47.024217 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:47.024223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:47.024264 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:12:47.026570 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:47.026582 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:47.026930 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:47.027149 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:47.028521 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.030482 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.032103 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.033094 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.033299 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.037393 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:47.037407 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:47.037428 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:47.037475 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:47.041882 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.042690 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.042699 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.042703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.042913 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.043312 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.043320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.043323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.044097 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:47.044326 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:47.044809 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 13:12:47.045143 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":428,"thread_start_us":290,"threads_started":1}
I20260619 13:12:47.053016 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:47.054415 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.053840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33946)
0619 13:12:47.054061 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.054211 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 13:12:47.054808 22821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
I20260619 13:12:47.056562 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.056174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33950)
0619 13:12:47.056323 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.056433 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:12:47.056787 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36721: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:12:47.076719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36721
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:12:47.106186 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } attempt: 1
I20260619 13:12:47.107386 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.106937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33952)
0619 13:12:47.107144 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.107237 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:12:47.108570 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.108085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33966)
0619 13:12:47.108281 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.108379 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 13:12:47.108790 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36721: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
I20260619 13:12:47.177062 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } attempt: 2
I20260619 13:12:47.178594 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.177890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33982)
0619 13:12:47.178197 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.178363 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:12:47.179735 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.179313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33986)
0619 13:12:47.179466 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.179577 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:12:47.179929 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36721: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
I20260619 13:12:47.271209 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } attempt: 3
I20260619 13:12:47.272529 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.271978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33992)
0619 13:12:47.272218 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.272343 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 13:12:47.273715 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.273225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:33998)
0619 13:12:47.273379 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.273464 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:12:47.273942 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36721: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
I20260619 13:12:47.417260 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } attempt: 4
I20260619 13:12:47.418623 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.418050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:34006)
0619 13:12:47.418364 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.418466 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260619 13:12:47.419716 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.419387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:34016)
0619 13:12:47.419519 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.419590 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:12:47.419940 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36721: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36721: connect: Connection refused (error 111)
W20260619 13:12:47.424574 22876 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:47.424962 22876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:47.425077 22876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:47.425208 22876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:47.432875 22876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:47.432971 22876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:47.433091 22876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:47.433164 22876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:47.433267 22876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:47.443699 22876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:47.445086 22876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:47.446463 22876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:47.458655 22882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:47.458688 22881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:47.458670 22884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:47.459630 22876 server_base.cc:1034] running on GCE node
I20260619 13:12:47.460459 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:47.462044 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:47.463296 22876 hybrid_clock.cc:648] HybridClock initialized: now 1781874767463235 us; error 41 us; skew 500 ppm
I20260619 13:12:47.463670 22876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:47.466958 22876 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:12:47.468677 22876 webserver.cc:460] Webserver started at http://127.0.97.61:40583/ using document root <none> and password file <none>
I20260619 13:12:47.469406 22876 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:47.469529 22876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:47.469992 22876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:47.473045 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data/instance:
uuid: "4be6a519a34546e8b57d9317f2eef9f9"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "8a71a49032f97cdc6539c73fed3b056c"
server_key_iv: "aa5990e678475f225ffefa1238be2a79"
server_key_version: "encryptionkey@0"
I20260619 13:12:47.473886 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal/instance:
uuid: "4be6a519a34546e8b57d9317f2eef9f9"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "8a71a49032f97cdc6539c73fed3b056c"
server_key_iv: "aa5990e678475f225ffefa1238be2a79"
server_key_version: "encryptionkey@0"
I20260619 13:12:47.479048 22876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:12:47.482460 22891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:47.483853 22876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:47.483995 22876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "4be6a519a34546e8b57d9317f2eef9f9"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "8a71a49032f97cdc6539c73fed3b056c"
server_key_iv: "aa5990e678475f225ffefa1238be2a79"
server_key_version: "encryptionkey@0"
I20260619 13:12:47.484174 22876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:47.503409 22876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:47.507738 22876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:47.507973 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:47.527422 22876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36721
I20260619 13:12:47.527458 22942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36721 every 8 connection(s)
I20260619 13:12:47.528811 22876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:12:47.532795 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:47.532935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22876
I20260619 13:12:47.533114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-1/wal/instance
I20260619 13:12:47.533727 388 external_mini_cluster.cc:1428] Setting key a05b8eba18d356f64f13ed15c7112f46
I20260619 13:12:47.542912 22943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 13:12:47.565541 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.546199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.0.97.62:39907)
0619 13:12:47.546392 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:47.546397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:47.548173 (+ 1776us) server_negotiation.cc:407] Connection header received
0619 13:12:47.550190 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:47.550197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:47.550287 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:47.550412 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:47.551929 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.552619 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.554234 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.554675 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.554818 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.558780 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:47.558785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:47.558788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:47.558822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:47.561816 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.562713 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.562722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.562728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.562899 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.563357 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.563365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.563372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.563845 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:47.564025 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:47.564772 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 13:12:47.565058 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:12:47.566181 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.546799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.0.1:55472)
0619 13:12:47.548003 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.548042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:47.549900 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:47.550659 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:47.550668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:47.551194 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:47.551750 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.551767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.552793 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.552797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.554076 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.554088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.554928 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.554936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.555174 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.555970 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:47.555993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:47.558515 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:47.561993 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.562007 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.562029 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.562546 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.563027 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.563031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.563033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.563208 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.563978 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:47.563984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:47.564351 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 13:12:47.564587 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 13:12:47.564980 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":424,"threads_started":1}
I20260619 13:12:47.568120 22943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:47.585043 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.569044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:34020)
0619 13:12:47.569235 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.569257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:47.569363 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:47.570519 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:47.570523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:47.570976 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:47.571120 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.571128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.572724 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.572732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.574347 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.574365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.575288 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.575300 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.575521 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.576178 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:47.576206 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:47.578918 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:47.582337 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.582343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.582347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.582740 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.583142 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.583148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.583151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.583276 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.583998 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:47.584004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:47.584171 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 13:12:47.584323 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:12:47.584635 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:12:47.585153 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.569331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.0.97.61:36721)
0619 13:12:47.570110 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:47.570116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:47.570142 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:12:47.570184 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:47.570190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:47.570369 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:47.570538 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:47.571339 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.572537 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.574577 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.575126 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.575239 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.579138 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:47.579150 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:47.579165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:47.579199 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:47.582178 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.582904 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.582910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.582912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.583029 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.583398 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.583404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.583407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.583871 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:47.584112 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:47.584479 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 13:12:47.584753 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":630,"thread_start_us":489,"threads_started":1}
I20260619 13:12:47.588171 22943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:47.589336 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.588847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47542)
0619 13:12:47.589005 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.589172 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 13:12:47.589632 22895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.591327 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.590954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47550)
0619 13:12:47.591143 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.591219 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:12:47.591535 22943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:12:47.595286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40167
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:12:47.640852 22943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 1
I20260619 13:12:47.642371 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.641900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47560)
0619 13:12:47.642112 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.642208 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:12:47.643455 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.643088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47568)
0619 13:12:47.643251 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.643340 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:12:47.643707 22943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.711937 22943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 2
I20260619 13:12:47.713224 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.712736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47574)
0619 13:12:47.712956 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.713060 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:12:47.714386 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.714008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47590)
0619 13:12:47.714169 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.714256 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 13:12:47.714596 22943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.719218 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } attempt: 5
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:47.734704 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.719927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:34036)
0619 13:12:47.720142 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.720163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:47.720307 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:47.720723 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:47.720727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:47.721108 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:47.721285 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.721293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.722908 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.722912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.723772 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:47.723783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.724776 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.724781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:47.724926 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.725723 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:47.725740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:47.728041 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:47.731946 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.731951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.731953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.732360 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.732769 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:47.732772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:47.732774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:47.732919 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:47.733698 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:47.733706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:47.733916 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 13:12:47.734100 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 13:12:47.734427 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:12:47.734831 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.720161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.0.97.61:36721)
0619 13:12:47.720361 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:47.720367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:47.720390 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:12:47.720473 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:47.720479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:47.720558 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:47.720684 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:47.721451 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.722735 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.723927 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:47.724617 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:47.724748 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:47.728259 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:47.728265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:47.728270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:47.728318 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:47.731792 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.732486 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.732491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.732495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.732649 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:47.733042 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:47.733049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:47.733053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:47.733522 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:47.733738 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:47.734281 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 13:12:47.734535 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:12:47.736030 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:47.737124 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.736724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47606)
0619 13:12:47.736889 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.736969 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 13:12:47.738124 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.737814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47620)
0619 13:12:47.737951 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.738033 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:12:47.738433 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.789741 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 1
I20260619 13:12:47.791010 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.790533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47628)
0619 13:12:47.790732 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.790834 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 13:12:47.792133 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.791782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47636)
0619 13:12:47.791948 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.792024 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 13:12:47.792431 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.805820 22943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 3
I20260619 13:12:47.807096 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.806550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47640)
0619 13:12:47.806836 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.806938 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 13:12:47.808320 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.807888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47646)
0619 13:12:47.808111 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.808190 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260619 13:12:47.808635 22943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.860697 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 2
I20260619 13:12:47.862049 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.861477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47660)
0619 13:12:47.861761 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.861877 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 13:12:47.863134 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.862780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47670)
0619 13:12:47.862943 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.863026 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:12:47.863353 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
W20260619 13:12:47.940279 22950 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:47.940677 22950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:47.940802 22950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:47.940943 22950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:47.948235 22950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:47.948377 22950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:47.948484 22950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:47.948572 22950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:47.948662 22950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:47.951862 22943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 4
I20260619 13:12:47.953044 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.952558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47684)
0619 13:12:47.952780 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.952881 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 13:12:47.954106 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.953779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47696)
0619 13:12:47.953929 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.954008 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:12:47.954315 22943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
I20260619 13:12:47.959318 22950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39907,127.0.97.61:36721,127.0.97.60:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:47.960654 22950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:47.962071 22950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:12:47.969554 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 3
I20260619 13:12:47.970721 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.970268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47698)
0619 13:12:47.970486 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.970573 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 13:12:47.971763 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:47.971449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47702)
0619 13:12:47.971602 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:47.971666 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:12:47.972002 22869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40167: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40167: connect: Connection refused (error 111)
W20260619 13:12:47.975888 22956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:47.975807 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:47.975901 22958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:47.981948 22950 server_base.cc:1034] running on GCE node
I20260619 13:12:47.982939 22950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:47.984547 22950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:47.985805 22950 hybrid_clock.cc:648] HybridClock initialized: now 1781874767985719 us; error 67 us; skew 500 ppm
I20260619 13:12:47.986164 22950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:47 dist-test-slave-q11v krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:47.989800 22950 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:12:47.991523 22950 webserver.cc:460] Webserver started at http://127.0.97.60:38253/ using document root <none> and password file <none>
I20260619 13:12:47.992316 22950 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:47.992455 22950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:47.992794 22950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:47.996083 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data/instance:
uuid: "fcf02e98a3584f31a8bf81c9b0cec82a"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "9d08346a8ec16c663a83578ea150703d"
server_key_iv: "23edcd677afda6d8b1011c0a4644f0b8"
server_key_version: "encryptionkey@0"
I20260619 13:12:47.996925 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal/instance:
uuid: "fcf02e98a3584f31a8bf81c9b0cec82a"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "9d08346a8ec16c663a83578ea150703d"
server_key_iv: "23edcd677afda6d8b1011c0a4644f0b8"
server_key_version: "encryptionkey@0"
I20260619 13:12:48.002681 22950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:12:48.006683 22965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:48.008364 22950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:12:48.008585 22950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "fcf02e98a3584f31a8bf81c9b0cec82a"
format_stamp: "Formatted at 2026-06-19 13:12:47 on dist-test-slave-q11v"
server_key: "9d08346a8ec16c663a83578ea150703d"
server_key_iv: "23edcd677afda6d8b1011c0a4644f0b8"
server_key_version: "encryptionkey@0"
I20260619 13:12:48.008841 22950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:48.030351 22950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:48.035485 22950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:48.035774 22950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:48.055685 22950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40167
I20260619 13:12:48.055702 23016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40167 every 8 connection(s)
I20260619 13:12:48.057093 22950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:12:48.060894 23017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:48.061908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 22950
I20260619 13:12:48.062126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874405339864-388-0/minicluster-data/master-2/wal/instance
I20260619 13:12:48.062678 388 external_mini_cluster.cc:1428] Setting key b7221e40a4eb464c10a97da48b7a5a17
I20260619 13:12:48.071110 23017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:48.088109 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.064677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.0.97.60:40167)
0619 13:12:48.065515 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.065527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.065574 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:12:48.067633 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.067656 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.068145 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.068561 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.070466 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.071821 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.073775 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.075021 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.075247 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.079646 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.079653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.079664 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.079698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.083556 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.084415 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.084422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.084429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.084591 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.085068 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.085072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.085074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.085659 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.085945 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.086378 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.086760 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":390,"threads_started":1}
I20260619 13:12:48.090397 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.073998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.0.97.62:39907)
0619 13:12:48.074541 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.074549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.075056 (+ 507us) server_negotiation.cc:407] Connection header received
0619 13:12:48.075311 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.075317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.075425 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.075532 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.077023 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.077874 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.079312 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.079822 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.079918 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.083913 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.083921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.083928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.083970 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.087256 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.088088 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.088093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.088097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.088286 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.088750 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.088757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.088762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.089048 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.089174 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.089755 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.089983 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260619 13:12:48.090418 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.074088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.0.1:55484)
0619 13:12:48.074902 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:48.074948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:48.075168 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:48.075640 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:48.075649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:48.076502 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:48.076853 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.076866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.078044 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.078052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.079157 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.079169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.080031 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.080038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.080262 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.080777 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:48.080804 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:48.083718 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:48.087385 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.087394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.087400 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.087946 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.088452 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.088457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.088460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.088614 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.089165 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:48.089171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:48.089403 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 13:12:48.089646 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 13:12:48.089966 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":472,"threads_started":1}
I20260619 13:12:48.092187 23017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:48.106174 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.093015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36721 (local address 127.0.0.1:34040)
0619 13:12:48.093273 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:48.093289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:48.093409 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:48.093760 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:48.093765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:48.094131 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:48.094269 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.094277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.095293 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.095301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.096418 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.096433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.097297 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.097306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.097534 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.098092 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:48.098118 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:48.100550 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:48.103658 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.103664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.103668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.104152 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.104492 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.104500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.104505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.104677 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.105278 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:48.105281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:48.105412 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 13:12:48.105559 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 13:12:48.105898 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 13:12:48.106398 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.092948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.0.97.61:36721)
0619 13:12:48.093124 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.093128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.093356 (+ 228us) server_negotiation.cc:407] Connection header received
0619 13:12:48.093517 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.093520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.093577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.093741 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.094413 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.095072 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.096563 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.097147 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.097239 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.100740 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.100748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.100754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.100797 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.103516 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.104262 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.104266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.104268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.104385 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.104782 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.104785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.104787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.105158 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.105315 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.105729 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.105970 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:12:48.107478 23017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:48.107945 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.091748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.97.62:39907)
0619 13:12:48.091902 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.091907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.091929 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:12:48.091967 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.091970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.092033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.092143 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.093690 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.094363 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.095755 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.096176 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.096272 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.101639 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.101647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.101655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.101702 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.104817 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.105566 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.105571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.105574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.105727 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.106154 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.106161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.106165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.106552 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.106823 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.107376 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.107570 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:48.122133 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.108328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.97.60:40167)
0619 13:12:48.108487 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.108492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.108542 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:12:48.108708 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.108711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.108762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.108874 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.109678 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.110604 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.112097 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.112725 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.112814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.116148 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.116153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.116156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.116182 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.119269 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.120061 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.120066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.120068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.120191 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.120609 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.120617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.120621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.120941 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.121092 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.121423 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.121736 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":50}
I20260619 13:12:48.122236 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.108167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47720)
0619 13:12:48.108461 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:48.108492 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:48.108607 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:48.108892 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:48.108896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:48.109319 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:48.109526 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.109539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.110738 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.110746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.111964 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.111976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.112908 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.112914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.113143 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.113719 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:48.113740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:48.115976 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:48.119398 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.119403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.119407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.119925 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.120359 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.120364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.120367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.120504 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.121053 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:48.121058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:48.121196 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 13:12:48.121340 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 13:12:48.121874 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":213}
I20260619 13:12:48.125264 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.109655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.0.97.61:36721)
0619 13:12:48.109819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.109825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.109907 (+ 82us) server_negotiation.cc:407] Connection header received
0619 13:12:48.110017 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.110022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.110090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.110189 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.111581 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.112490 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.114236 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.114630 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.114725 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.119452 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.119458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.119464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.119507 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.122505 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.123335 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.123343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.123349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.123526 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.123980 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.123984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.123990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.124265 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.124467 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.124792 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.124961 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:12:48.128580 23017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: Bootstrap starting.
I20260619 13:12:48.130246 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:48.132473 23017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:48.133879 23017 log.cc:824] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:48.136930 23017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: No bootstrap required, opened a new log
I20260619 13:12:48.133826 22844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "697a0f8e71c44c87a44416686affd9ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55498
I20260619 13:12:48.144229 23017 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.144613 23017 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf02e98a3584f31a8bf81c9b0cec82a, State: Initialized, Role: FOLLOWER
I20260619 13:12:48.145083 23017 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.146219 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.146643 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.147657 23017 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:48.149228 22869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 4
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 13:12:48.166527 23035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:12:48.166658 23035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:12:48.166972 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.150091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.0.97.60:40167)
0619 13:12:48.150286 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.150295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.150340 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:12:48.150538 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.150546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.150649 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.150795 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.151731 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.153218 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.155049 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.156071 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.156233 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.160129 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.160137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.160143 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.160186 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.163630 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.164483 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.164489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.164492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.164655 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.165157 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.165163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.165168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.165665 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.165824 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.166394 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.166643 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 13:12:48.167001 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.150076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47734)
0619 13:12:48.150250 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:48.150271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:48.150394 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:48.150824 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:48.150830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:48.151343 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:48.151573 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.151587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.153402 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.153407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.154897 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.154912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.156233 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.156240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.156438 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.157161 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:48.157178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:48.159927 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:48.163767 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.163774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.163780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.164326 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.164819 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.164824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.164827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.165022 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.165931 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:48.165936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:48.166117 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 13:12:48.166271 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 13:12:48.166600 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 13:12:48.172513 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: Bootstrap starting.
I20260619 13:12:48.163264 22950 master_runner.cc:186] Error getting master registration for 127.0.97.62:39907: OK, instance_id { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" instance_seqno: 1781874767001904 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:12:48.173897 22950 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:12:48.176651 22869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:48.178102 22869 log.cc:824] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:48.181314 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: No bootstrap required, opened a new log
I20260619 13:12:48.186568 22869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.186923 22869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697a0f8e71c44c87a44416686affd9ba, State: Initialized, Role: FOLLOWER
I20260619 13:12:48.187481 22869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.188601 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.189065 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.189818 22869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:48.189952 22844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:12:48.190171 22844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:48.193727 22844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.195425 22844 leader_election.cc:288] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [CANDIDATE]: Term 1 election: Requested vote from peers 4be6a519a34546e8b57d9317f2eef9f9 (127.0.97.61:36721), fcf02e98a3584f31a8bf81c9b0cec82a (127.0.97.60:40167)
I20260619 13:12:48.197014 22992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697a0f8e71c44c87a44416686affd9ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a"
I20260619 13:12:48.197897 22992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:48.202874 22992 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697a0f8e71c44c87a44416686affd9ba in term 1.
I20260619 13:12:48.203943 22821 leader_election.cc:302] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697a0f8e71c44c87a44416686affd9ba, fcf02e98a3584f31a8bf81c9b0cec82a; no voters:
I20260619 13:12:48.204664 23038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:48.216480 23038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [term 1 LEADER]: Becoming Leader. State: Replica: 697a0f8e71c44c87a44416686affd9ba, State: Running, Role: LEADER
I20260619 13:12:48.217437 23038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.226608 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697a0f8e71c44c87a44416686affd9ba. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.227335 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:48.233508 23052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 13:12:48.241410 22823 tablet.cc:2369] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:48.252663 23052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:48.253530 22943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } attempt: 5
I20260619 13:12:48.262908 23061 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:12:48.280023 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.226791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.97.61:36721)
0619 13:12:48.226965 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.226982 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.239456 (+ 12474us) server_negotiation.cc:407] Connection header received
0619 13:12:48.240621 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.240628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.240710 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.240783 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.248044 (+ 7261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.249219 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.259399 (+ 10180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.260135 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.260273 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.272389 (+ 12116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.272398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.272405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.272456 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.276230 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.277063 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.277069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.277073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.277243 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.277761 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.277770 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.277776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.278221 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.278465 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.279102 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.279733 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 13:12:48.285094 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.225056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.0.97.62:39907)
0619 13:12:48.225262 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.225268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.239320 (+ 14052us) server_negotiation.cc:407] Connection header received
0619 13:12:48.240156 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.240163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.240253 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.240336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.249564 (+ 9228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.250742 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.260619 (+ 9877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.261342 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.261448 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.271346 (+ 9898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.271356 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.271361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.271407 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.276281 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.282408 (+ 6127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.282414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.282419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.282616 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.283081 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.283086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.283090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.283473 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.283824 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.284372 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.284658 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 13:12:48.287998 22971 tablet.cc:2369] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:48.289556 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.251740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.97.60:40167)
0619 13:12:48.251941 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.251947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.257787 (+ 5840us) server_negotiation.cc:407] Connection header received
0619 13:12:48.269988 (+ 12201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.269995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.270102 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.270188 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.272837 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.274075 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.276080 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.276769 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.276893 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.280181 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.280189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.280196 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.280239 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.285679 (+ 5440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.286493 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.286502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.286508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.286673 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.287172 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.287180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.287184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.287537 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.287811 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.288925 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.289165 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:12:48.300073 22992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [term 1 FOLLOWER]: Refusing update from remote peer 697a0f8e71c44c87a44416686affd9ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 19 13:12:48 dist-test-slave-q11v krb5kdc[22789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:12:48.303412 23044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:48.268093 22802 master_runner.cc:186] Error getting master registration for 127.0.97.61:36721: OK, instance_id { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" instance_seqno: 1781874767522077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:12:48.309466 22802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:12:48.310781 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.254781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40167 (local address 127.0.0.1:47750)
0619 13:12:48.255366 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:48.255435 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:48.255586 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:48.278839 (+ 23253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:48.278845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:48.279391 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:48.279575 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.279588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.294857 (+ 15269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.294864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.296343 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:48.296365 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.297412 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.297419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:48.297688 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.298592 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:48.298618 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:48.301459 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:48.306345 (+ 4886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.306351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.306354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.306914 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.308120 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:48.308125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:48.308129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:48.308301 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:48.309714 (+ 1413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:48.309721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:48.309907 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 13:12:48.310064 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 13:12:48.310375 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260619 13:12:48.311942 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:48.255834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.0.97.60:40167)
0619 13:12:48.278377 (+ 22543us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.278383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.278407 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:12:48.278455 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.278460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.278561 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.280524 (+ 1963us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.280839 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.293740 (+ 12901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.296564 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:48.297238 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:48.297368 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:48.302109 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:48.302116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:48.302120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:48.302168 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:48.306183 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.307104 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.307109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.307112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.307786 (+ 674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:48.308442 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:48.308447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:48.308450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:48.308820 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:48.311159 (+ 2339us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:48.311297 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 13:12:48.311532 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22384,"thread_start_us":22791,"threads_started":1}
I20260619 13:12:48.319999 22943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: Bootstrap starting.
I20260619 13:12:48.324782 22943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:48.326413 22943 log.cc:824] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:48.332695 22943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: No bootstrap required, opened a new log
I20260619 13:12:48.340379 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697a0f8e71c44c87a44416686affd9ba. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.340755 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.349376 23062 mvcc.cc:205] Tried to move back new op lower bound from 7298559050931712000 to 7298559050627555328. Current Snapshot: MvccSnapshot[applied={T|T < 7298559050931712000}]
I20260619 13:12:48.350005 23063 mvcc.cc:205] Tried to move back new op lower bound from 7298559050931712000 to 7298559050627555328. Current Snapshot: MvccSnapshot[applied={T|T < 7298559050931712000}]
I20260619 13:12:48.352819 22943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.353325 22943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be6a519a34546e8b57d9317f2eef9f9, State: Initialized, Role: FOLLOWER
I20260619 13:12:48.354239 22943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } }
I20260619 13:12:48.358896 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.359282 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:48.359759 23052 catalog_manager.cc:1347] Generated new cluster ID: 1e1a4607a6264ad4a215f06a004c8478
I20260619 13:12:48.359874 23052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:48.368795 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.369117 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.374300 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.374779 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.375490 22943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:48.375552 22918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697a0f8e71c44c87a44416686affd9ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4be6a519a34546e8b57d9317f2eef9f9"
I20260619 13:12:48.376036 22918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:48.379650 22918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697a0f8e71c44c87a44416686affd9ba in term 1.
I20260619 13:12:48.382568 22917 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [term 1 FOLLOWER]: Refusing update from remote peer 697a0f8e71c44c87a44416686affd9ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:48.384095 23038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:48.435060 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697a0f8e71c44c87a44416686affd9ba. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.435557 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.435040 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.435889 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:48.466586 23083 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:12:48.483500 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "697a0f8e71c44c87a44416686affd9ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697a0f8e71c44c87a44416686affd9ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } peers { permanent_uuid: "4be6a519a34546e8b57d9317f2eef9f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36721 } } peers { permanent_uuid: "fcf02e98a3584f31a8bf81c9b0cec82a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40167 } } }
I20260619 13:12:48.483865 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:48.489689 23052 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:48.491667 23052 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:48.514020 23052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 697a0f8e71c44c87a44416686affd9ba: Generated new TSK 0
I20260619 13:12:48.515249 23052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:12:48.621403 23021 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.610970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.97.62:39907)
0619 13:12:48.611177 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.611182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.617980 (+ 6798us) server_negotiation.cc:407] Connection header received
0619 13:12:48.619591 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.619599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.619693 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.619779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.621118 (+ 1339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41594: BlockingRecv error: recv got EOF from 127.0.0.1:41594 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 13:12:48.625757 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.610696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.97.61:36721)
0619 13:12:48.610893 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.610900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.623303 (+ 12403us) server_negotiation.cc:407] Connection header received
0619 13:12:48.624017 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.624023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.624123 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.624214 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.625482 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35038: BlockingRecv error: recv got EOF from 127.0.0.1:35038 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 13:12:48.626401 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.618771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.97.60:40167)
0619 13:12:48.619040 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.619050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.623635 (+ 4585us) server_negotiation.cc:407] Connection header received
0619 13:12:48.624382 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.624388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.624502 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.624585 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.625519 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41700: BlockingRecv error: recv got EOF from 127.0.0.1:41700 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 13:12:48.634610 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.632134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.0.97.61:36721)
0619 13:12:48.632337 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.632343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.632852 (+ 509us) server_negotiation.cc:407] Connection header received
0619 13:12:48.633133 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.633141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.633271 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.633413 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.634336 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35048: BlockingRecv error: recv got EOF from 127.0.0.1:35048 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 13:12:48.641587 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.639338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.97.61:36721)
0619 13:12:48.639517 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.639524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.640007 (+ 483us) server_negotiation.cc:407] Connection header received
0619 13:12:48.640245 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.640251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.640358 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.640472 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.641355 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35060: BlockingRecv error: recv got EOF from 127.0.0.1:35060 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 13:12:48.649528 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.646995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.0.97.61:36721)
0619 13:12:48.647196 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.647203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.647832 (+ 629us) server_negotiation.cc:407] Connection header received
0619 13:12:48.648072 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.648078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.648183 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.648299 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.649256 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35066: BlockingRecv error: recv got EOF from 127.0.0.1:35066 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 13:12:48.659909 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.657429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.97.61:36721)
0619 13:12:48.657671 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.657677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.658298 (+ 621us) server_negotiation.cc:407] Connection header received
0619 13:12:48.658551 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.658557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.658657 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.658776 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.659673 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35078: BlockingRecv error: recv got EOF from 127.0.0.1:35078 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 13:12:48.674240 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.671632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.97.61:36721)
0619 13:12:48.671841 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.671847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.672450 (+ 603us) server_negotiation.cc:407] Connection header received
0619 13:12:48.672676 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.672683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.672785 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.672907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.673986 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35082: BlockingRecv error: recv got EOF from 127.0.0.1:35082 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 13:12:48.701082 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.698572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.97.61:36721)
0619 13:12:48.698780 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.698787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.699428 (+ 641us) server_negotiation.cc:407] Connection header received
0619 13:12:48.699664 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.699670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.699779 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.699906 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.700840 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 13:12:48.740439 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.737987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.97.61:36721)
0619 13:12:48.738187 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.738193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.738766 (+ 573us) server_negotiation.cc:407] Connection header received
0619 13:12:48.739023 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.739029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.739124 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.739226 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.740149 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 13:12:48.809933 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.807686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.97.61:36721)
0619 13:12:48.807951 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.807961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.808359 (+ 398us) server_negotiation.cc:407] Connection header received
0619 13:12:48.808632 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.808637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.808716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.808892 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.809736 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35112: BlockingRecv error: recv got EOF from 127.0.0.1:35112 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 13:12:48.945130 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:48.943252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.0.97.61:36721)
0619 13:12:48.943526 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:48.943534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:48.943732 (+ 198us) server_negotiation.cc:407] Connection header received
0619 13:12:48.943934 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:48.943939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:48.944018 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:48.944117 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:48.944879 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35128: BlockingRecv error: recv got EOF from 127.0.0.1:35128 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 13:12:49.168912 23035 catalog_manager.cc:1259] Loaded cluster ID: 1e1a4607a6264ad4a215f06a004c8478
I20260619 13:12:49.169018 23035 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: loading cluster ID for follower catalog manager: success
I20260619 13:12:49.171689 23035 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: acquiring CA information for follower catalog manager: success
I20260619 13:12:49.173589 23035 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcf02e98a3584f31a8bf81c9b0cec82a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:12:49.209399 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:49.207527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.97.61:36721)
0619 13:12:49.207769 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.207776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.207930 (+ 154us) server_negotiation.cc:407] Connection header received
0619 13:12:49.208171 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.208179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.208256 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.208381 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:49.209209 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:12:49.471570 23083 catalog_manager.cc:1259] Loaded cluster ID: 1e1a4607a6264ad4a215f06a004c8478
I20260619 13:12:49.471692 23083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: loading cluster ID for follower catalog manager: success
I20260619 13:12:49.474515 23083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: acquiring CA information for follower catalog manager: success
I20260619 13:12:49.476573 23083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4be6a519a34546e8b57d9317f2eef9f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:12:49.733839 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:49.725890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.97.61:36721)
0619 13:12:49.726809 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.726814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.726832 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:12:49.726898 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.726901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.727055 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.727173 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:49.727715 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.728641 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.730751 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.731258 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.731368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:49.732175 (+ 807us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:49.732876 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:49.733053 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:49.733176 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:12:49.733267 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":659,"threads_started":1}
I20260619 13:12:49.748569 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:49.740909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.0.97.60:40167)
0619 13:12:49.741462 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.741469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.741914 (+ 445us) server_negotiation.cc:407] Connection header received
0619 13:12:49.742140 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.742146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.742375 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.742582 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:49.743045 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.744173 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.745784 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.746422 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.746562 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:49.747076 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:49.747927 (+ 851us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:49.748083 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:49.748136 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 13:12:49.748229 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
I20260619 13:12:49.765971 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:49.755845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.0.97.61:36721)
0619 13:12:49.756328 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.756334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.758428 (+ 2094us) server_negotiation.cc:407] Connection header received
0619 13:12:49.758741 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.758748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.758841 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.758910 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:49.759746 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.760810 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.763512 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.764195 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.764316 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:49.765029 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:49.765152 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:49.765228 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:49.765501 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:12:49.765587 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260619 13:12:49.766445 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:49.756017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.97.62:39907)
0619 13:12:49.758025 (+ 2008us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.758034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.758103 (+ 69us) server_negotiation.cc:407] Connection header received
0619 13:12:49.758234 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.758241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.758481 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.759252 (+ 771us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:49.759440 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.760577 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.763078 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.763707 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.764334 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:49.764541 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:49.765069 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:49.765574 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:49.765648 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 13:12:49.765734 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":1759,"threads_started":1}
I20260619 13:12:49.770758 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:49.758638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.97.60:40167)
0619 13:12:49.758829 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:49.758836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:49.763218 (+ 4382us) server_negotiation.cc:407] Connection header received
0619 13:12:49.763501 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:49.763507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:49.763616 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:49.763682 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:49.764830 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.765915 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.768722 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:49.769333 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:49.769481 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:49.769988 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:49.770137 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:49.770239 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:49.770363 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 13:12:49.770453 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:12:49.771615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22802
I20260619 13:12:49.772679 22864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:49.908834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22802
W20260619 13:12:49.930006 22966 connection.cc:535] server connection from 127.0.0.1:47734 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:12:49.930347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22876
I20260619 13:12:49.931473 22938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:50.038810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22876
I20260619 13:12:50.060695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 22950
I20260619 13:12:50.061895 23012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:50.175542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 22950
2026-06-19T13:12:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4983 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:12:50 dist-test-slave-q11v krb5kdc[23170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:12:50 dist-test-slave-q11v krb5kdc[23170](info): set up 2 sockets
Jun 19 13:12:50 dist-test-slave-q11v krb5kdc[23170](info): commencing operation
krb5kdc: starting...
W20260619 13:12:51.314325 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/test-user.keytab.
Jun 19 13:12:51 dist-test-slave-q11v krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:12:51.582823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40763
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:12:51.946280 23183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:51.946667 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:51.946803 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:51.946946 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:12:51.954619 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:51.954723 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:51.954864 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:51.954950 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:51.955067 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:51.966075 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:51.967521 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:51.968971 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:51.981437 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:51.981403 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:51.981384 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:51.982728 23183 server_base.cc:1034] running on GCE node
I20260619 13:12:51.983501 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:51.984907 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:51.986125 23183 hybrid_clock.cc:648] HybridClock initialized: now 1781874771986066 us; error 29 us; skew 500 ppm
I20260619 13:12:51.986495 23183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:51 dist-test-slave-q11v krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:51.989681 23183 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:12:51.991079 23183 webserver.cc:460] Webserver started at http://127.0.97.62:33179/ using document root <none> and password file <none>
I20260619 13:12:51.991791 23183 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:51.991925 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:51.992245 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:51.995232 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data/instance:
uuid: "23c87ca8caee4944b751f9e6698cf985"
format_stamp: "Formatted at 2026-06-19 13:12:51 on dist-test-slave-q11v"
server_key: "f79499d1809a839f878994b2c3f4596a"
server_key_iv: "705bc23b90a7d078b870ceee7039b62a"
server_key_version: "encryptionkey@0"
I20260619 13:12:51.996022 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance:
uuid: "23c87ca8caee4944b751f9e6698cf985"
format_stamp: "Formatted at 2026-06-19 13:12:51 on dist-test-slave-q11v"
server_key: "f79499d1809a839f878994b2c3f4596a"
server_key_iv: "705bc23b90a7d078b870ceee7039b62a"
server_key_version: "encryptionkey@0"
I20260619 13:12:52.001190 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:12:52.005021 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:52.006423 23183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:52.006634 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal
uuid: "23c87ca8caee4944b751f9e6698cf985"
format_stamp: "Formatted at 2026-06-19 13:12:51 on dist-test-slave-q11v"
server_key: "f79499d1809a839f878994b2c3f4596a"
server_key_iv: "705bc23b90a7d078b870ceee7039b62a"
server_key_version: "encryptionkey@0"
I20260619 13:12:52.006891 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:52.030932 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:52.035519 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:52.035830 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:52.055323 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40763
I20260619 13:12:52.055343 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40763 every 8 connection(s)
I20260619 13:12:52.056766 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:12:52.059558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 23183
I20260619 13:12:52.059764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-0/wal/instance
I20260619 13:12:52.060303 388 external_mini_cluster.cc:1428] Setting key ddbeb3fbaab0a9b5ada3be98e9de7340
I20260619 13:12:52.061348 23250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:52.071664 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:52 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 13:12:52.095315 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.075050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.0.97.62:40763)
0619 13:12:52.076768 (+ 1718us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:52.076776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:52.076824 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:12:52.078465 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:52.078471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:52.078731 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:52.078954 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:52.080287 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.081511 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.083035 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.084080 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.084340 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.087733 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:52.087741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:52.087755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:52.087788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:52.090708 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.091674 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.091682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.091684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.091912 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.092427 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.092431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.092433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.092996 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:52.093166 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:52.093892 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 13:12:52.094200 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":1485,"thread_start_us":1287,"threads_started":1}
I20260619 13:12:52.095377 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.075045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40763 (local address 127.0.0.1:44274)
0619 13:12:52.075933 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.075989 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:52.078054 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:52.078967 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:52.078973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:52.079714 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:52.080065 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.080083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.081718 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.081725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.082815 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.082827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.084334 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.084339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.084468 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.085169 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:52.085193 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:52.087485 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:52.090862 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.090877 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.090904 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.091512 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.092049 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.092058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.092065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.092288 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.093149 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:52.093160 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:52.093451 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 13:12:52.093760 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 13:12:52.094395 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":78,"thread_start_us":389,"threads_started":1}
I20260619 13:12:52.098516 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:52.099694 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.099266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54834)
0619 13:12:52.099448 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.099572 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 13:12:52.100121 23201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
I20260619 13:12:52.101940 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.101491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54842)
0619 13:12:52.101716 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.101801 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 13:12:52.102255 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33301: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:12:52.122871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33301
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:12:52.151639 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } attempt: 1
I20260619 13:12:52.152953 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.152438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54852)
0619 13:12:52.152658 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.152791 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:12:52.154145 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.153777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54856)
0619 13:12:52.153925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.154008 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:12:52.154409 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33301: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
I20260619 13:12:52.222672 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } attempt: 2
I20260619 13:12:52.224015 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.223479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54868)
0619 13:12:52.223743 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.223850 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:12:52.225086 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.224710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54880)
0619 13:12:52.224856 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.224948 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:12:52.225275 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33301: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
I20260619 13:12:52.316637 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } attempt: 3
I20260619 13:12:52.317866 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.317317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54892)
0619 13:12:52.317614 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.317721 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 13:12:52.318903 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.318528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54900)
0619 13:12:52.318675 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.318748 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:12:52.319072 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33301: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
W20260619 13:12:52.460376 23257 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:52.460826 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:52.460965 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:52.461108 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 13:12:52.462307 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } attempt: 4
I20260619 13:12:52.463593 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.462959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54906)
0619 13:12:52.463309 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.463419 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 13:12:52.464669 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.464267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54908)
0619 13:12:52.464424 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.464515 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:12:52.464901 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33301: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33301: connect: Connection refused (error 111)
W20260619 13:12:52.469331 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:52.469416 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:52.469565 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:52.469671 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:52.469785 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:52.480053 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:52.481416 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:52.482865 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:52.494638 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:52.494577 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:52.494540 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:52.495649 23257 server_base.cc:1034] running on GCE node
I20260619 13:12:52.496433 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:52.497917 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:52.499166 23257 hybrid_clock.cc:648] HybridClock initialized: now 1781874772499091 us; error 50 us; skew 500 ppm
I20260619 13:12:52.499500 23257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:52 dist-test-slave-q11v krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:52.502799 23257 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:12:52.504280 23257 webserver.cc:460] Webserver started at http://127.0.97.61:37761/ using document root <none> and password file <none>
I20260619 13:12:52.505004 23257 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:52.505095 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:52.505360 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:52.508452 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data/instance:
uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77"
format_stamp: "Formatted at 2026-06-19 13:12:52 on dist-test-slave-q11v"
server_key: "0bfa3dd53fca261b513fe6d5c810f037"
server_key_iv: "fa5988ccc09314d06449daf22be10a38"
server_key_version: "encryptionkey@0"
I20260619 13:12:52.509197 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal/instance:
uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77"
format_stamp: "Formatted at 2026-06-19 13:12:52 on dist-test-slave-q11v"
server_key: "0bfa3dd53fca261b513fe6d5c810f037"
server_key_iv: "fa5988ccc09314d06449daf22be10a38"
server_key_version: "encryptionkey@0"
I20260619 13:12:52.514602 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:12:52.517910 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:52.519230 23257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:12:52.519375 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal
uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77"
format_stamp: "Formatted at 2026-06-19 13:12:52 on dist-test-slave-q11v"
server_key: "0bfa3dd53fca261b513fe6d5c810f037"
server_key_iv: "fa5988ccc09314d06449daf22be10a38"
server_key_version: "encryptionkey@0"
I20260619 13:12:52.519552 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:52.552835 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:52.557770 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:52.558038 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:52.577535 23257 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33301
I20260619 13:12:52.577562 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33301 every 8 connection(s)
I20260619 13:12:52.578992 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:12:52.582847 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:52.588547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 23257
I20260619 13:12:52.588737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-1/wal/instance
I20260619 13:12:52.589267 388 external_mini_cluster.cc:1428] Setting key 21d017ff15e00c317b15ccffe23ada1d
I20260619 13:12:52.590238 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:52 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:52.615631 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.593844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.97.62:40763)
0619 13:12:52.594067 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:52.594073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:52.595261 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 13:12:52.598261 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:52.598265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:52.598343 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:52.598452 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:52.600468 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.601663 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.603346 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.604127 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.604272 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.608216 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:52.608225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:52.608233 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:52.608282 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:52.612145 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.612933 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.612942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.612949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.613103 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.613642 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.613650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.613655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.614066 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:52.614244 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:52.614948 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 13:12:52.615228 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:12:52.616384 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.594227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40763 (local address 127.0.0.1:44276)
0619 13:12:52.595072 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.595130 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:52.597944 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:52.598761 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:52.598775 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:52.599628 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:52.600265 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.600283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.601870 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.601874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.603158 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.603171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.604375 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.604380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.604591 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.605382 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:52.605406 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:52.607951 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:52.612300 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.612308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.612323 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.612775 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.613228 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.613231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.613234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.613505 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.614196 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:52.614202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:52.614553 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 13:12:52.614821 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 13:12:52.615229 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 13:12:52.618237 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 13:12:52 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:52.635545 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.619261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.0.97.61:33301)
0619 13:12:52.620003 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:52.620012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:52.620046 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:12:52.620088 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:52.620094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:52.620341 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:52.620463 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:52.621238 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.623096 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.624477 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.625414 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.625552 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.628914 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:52.628931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:52.628953 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:52.628995 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:52.632339 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.633206 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.633212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.633215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.633339 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.633817 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.633821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.633823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.634222 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:52.634426 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:52.634880 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 13:12:52.635218 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":567,"thread_start_us":423,"threads_started":1}
I20260619 13:12:52.635578 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.619051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54922)
0619 13:12:52.619255 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.619274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:52.619415 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:52.620517 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:52.620521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:52.620925 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:52.621062 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.621070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.623269 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.623273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.624325 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.624337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.625552 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.625556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.625802 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.626424 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:52.626442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:52.628676 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:52.632482 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.632489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.632496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.633064 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.633447 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.633455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.633461 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.633695 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.634400 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:52.634408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:52.634580 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 13:12:52.634736 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 13:12:52.635068 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":85}
I20260619 13:12:52.638219 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:52.639336 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.638905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33692)
0619 13:12:52.639061 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.639191 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 13:12:52.639767 23273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:52.641361 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.641000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33702)
0619 13:12:52.641170 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.641253 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 13:12:52.641700 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:12:52.655081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
/tmp/dist-test-taskwPkIj_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39647
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:12:52.691030 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 1
I20260619 13:12:52.692265 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.691799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33708)
0619 13:12:52.692015 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.692107 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:12:52.693426 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.692986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33714)
0619 13:12:52.693208 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.693313 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 13:12:52.693655 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:52.761902 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 2
I20260619 13:12:52.763154 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.762676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33718)
0619 13:12:52.762901 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.762992 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 13:12:52.764176 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } attempt: 5
I20260619 13:12:52.764416 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.764020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33730)
0619 13:12:52.764180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.764267 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:12:52.764761 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Jun 19 13:12:52 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:52.779218 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.764861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.0.97.61:33301)
0619 13:12:52.764997 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:52.765001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:52.765117 (+ 116us) server_negotiation.cc:407] Connection header received
0619 13:12:52.765317 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:52.765320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:52.765380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:52.765466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:52.766466 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.767209 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.768929 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.769389 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.769495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.773559 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:52.773564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:52.773567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:52.773650 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:52.776432 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.777146 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.777152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.777155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.777288 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:52.777736 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:52.777742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:52.777746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:52.778058 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:52.778223 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:52.778665 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 13:12:52.778896 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 13:12:52.779229 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.764756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54938)
0619 13:12:52.765024 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.765042 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:52.765173 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:52.765536 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:52.765541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:52.766072 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:52.766303 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.766317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.767377 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.767382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.768744 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:52.768761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:52.769569 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:52.769577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:52.769879 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:52.770659 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:52.770691 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:52.773336 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:52.776596 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.776602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.776607 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.777013 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.777404 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:52.777410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:52.777414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:52.777562 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:52.778193 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:52.778211 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:52.778376 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 13:12:52.778531 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:12:52.778843 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 13:12:52.780539 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:52.781739 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.781228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33746)
0619 13:12:52.781440 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.781522 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 13:12:52.782697 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.782372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33750)
0619 13:12:52.782516 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.782599 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:12:52.782991 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:52.834277 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 1
I20260619 13:12:52.835608 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.835118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33762)
0619 13:12:52.835338 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.835440 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 13:12:52.836663 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.836372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33770)
0619 13:12:52.836504 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.836568 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:12:52.836932 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:52.855998 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 3
I20260619 13:12:52.857419 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.856851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33782)
0619 13:12:52.857129 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.857222 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 13:12:52.858659 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.858185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33792)
0619 13:12:52.858455 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.858533 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260619 13:12:52.858913 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:52.905182 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 2
I20260619 13:12:52.906589 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.906003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33796)
0619 13:12:52.906218 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.906329 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:12:52.907773 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:52.907405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33798)
0619 13:12:52.907572 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:52.907640 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:12:52.907971 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
W20260619 13:12:53.000821 23331 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:12:53.001235 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:12:53.001371 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:12:53.001514 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 13:12:53.002120 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 4
I20260619 13:12:53.003373 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.002871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33810)
0619 13:12:53.003115 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.003210 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:12:53.004418 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.004078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33818)
0619 13:12:53.004231 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.004312 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:12:53.004628 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
W20260619 13:12:53.010128 23331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:12:53.010258 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:12:53.010381 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:12:53.010468 23331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:12:53.010597 23331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:12:53.014165 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 3
I20260619 13:12:53.015304 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.014825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33820)
0619 13:12:53.015041 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.015137 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 13:12:53.016347 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33832)
0619 13:12:53.016164 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.016231 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 13:12:53.016561 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39647: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39647: connect: Connection refused (error 111)
I20260619 13:12:53.021538 23331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40763,127.0.97.61:33301,127.0.97.60:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:12:53.023036 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:12:53.024427 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:12:53.036496 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:53.036391 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:12:53.036451 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:12:53.037261 23331 server_base.cc:1034] running on GCE node
I20260619 13:12:53.038213 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:12:53.039836 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:12:53.041109 23331 hybrid_clock.cc:648] HybridClock initialized: now 1781874773041047 us; error 44 us; skew 500 ppm
I20260619 13:12:53.041508 23331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:12:53.044826 23331 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:12:53.046468 23331 webserver.cc:460] Webserver started at http://127.0.97.60:44883/ using document root <none> and password file <none>
I20260619 13:12:53.047180 23331 fs_manager.cc:359] Metadata directory not provided
I20260619 13:12:53.047272 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:12:53.047562 23331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:12:53.050710 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data/instance:
uuid: "d12becb655ab43e593f5b8c2f9c7ad01"
format_stamp: "Formatted at 2026-06-19 13:12:53 on dist-test-slave-q11v"
server_key: "cace573331b83f07ca0aad8fbf6f9106"
server_key_iv: "c4df236bf4de6361cc1d104decfc777a"
server_key_version: "encryptionkey@0"
I20260619 13:12:53.051504 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal/instance:
uuid: "d12becb655ab43e593f5b8c2f9c7ad01"
format_stamp: "Formatted at 2026-06-19 13:12:53 on dist-test-slave-q11v"
server_key: "cace573331b83f07ca0aad8fbf6f9106"
server_key_iv: "c4df236bf4de6361cc1d104decfc777a"
server_key_version: "encryptionkey@0"
I20260619 13:12:53.057291 23331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:12:53.061323 23346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:12:53.062829 23331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:12:53.063035 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal
uuid: "d12becb655ab43e593f5b8c2f9c7ad01"
format_stamp: "Formatted at 2026-06-19 13:12:53 on dist-test-slave-q11v"
server_key: "cace573331b83f07ca0aad8fbf6f9106"
server_key_iv: "c4df236bf4de6361cc1d104decfc777a"
server_key_version: "encryptionkey@0"
I20260619 13:12:53.063268 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:12:53.076395 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:12:53.080602 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:12:53.080892 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:12:53.099953 23331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39647
I20260619 13:12:53.099960 23397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39647 every 8 connection(s)
I20260619 13:12:53.101496 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:12:53.105376 23398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:12:53.111389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu as pid 23331
I20260619 13:12:53.111636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwPkIj_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874405339864-388-0/minicluster-data/master-2/wal/instance
I20260619 13:12:53.112164 388 external_mini_cluster.cc:1428] Setting key e0e47d191b92152de02087a59545bb2c
I20260619 13:12:53.112244 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:12:53.133387 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.115202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.0.97.62:40763)
0619 13:12:53.115440 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.115447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.116297 (+ 850us) server_negotiation.cc:407] Connection header received
0619 13:12:53.117883 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.117894 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.117962 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.118048 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.119694 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.120416 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.122427 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.122959 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.123125 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.126923 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.126928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.126932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.126971 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.130004 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.130823 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.130827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.130830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.130982 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.131432 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.131439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.131447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.131827 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.132007 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.132923 (+ 916us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.133111 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:12:53.134963 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.115211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40763 (local address 127.0.0.1:44278)
0619 13:12:53.116108 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.116166 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:53.117522 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:53.118161 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:53.118168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:53.119043 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:53.119496 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.119526 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.120591 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.120598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.122274 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.122287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.123237 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.123246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.123455 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.124342 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:53.124378 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:53.126687 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:53.130148 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.130160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.130180 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.130682 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.131097 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.131103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.131108 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.131293 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.131980 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:53.131987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:53.132436 (+ 449us) client_negotiation.cc:769] Sending connection context
0619 13:12:53.132784 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 13:12:53.133375 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1,"thread_start_us":412,"threads_started":1}
I20260619 13:12:53.134939 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.114172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.0.97.60:39647)
0619 13:12:53.115065 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.115082 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.115137 (+ 55us) server_negotiation.cc:407] Connection header received
0619 13:12:53.117522 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.117555 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.117959 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.118178 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.119565 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.121259 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.122644 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.123366 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.123530 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.128028 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.128037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.128051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.128084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.131466 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.132322 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.132332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.132335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.132501 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.132894 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.132899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.132902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.133345 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.133696 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.134112 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.134361 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":430,"threads_started":1}
I20260619 13:12:53.136693 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:12:53.150764 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.137616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33301 (local address 127.0.0.1:54948)
0619 13:12:53.137907 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.137923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:53.138013 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:53.138404 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:53.138411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:53.138896 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:53.139057 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.139065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.140059 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.140064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.140963 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.140974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.141814 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.141819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.142011 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.142657 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:53.142674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:53.144631 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:53.148221 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.148227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.148231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.148619 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.149021 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.149026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.149030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.149186 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.149783 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:53.149788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:53.149943 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 13:12:53.150085 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 13:12:53.150385 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260619 13:12:53.150792 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.137544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.0.97.61:33301)
0619 13:12:53.137800 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.137807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.138004 (+ 197us) server_negotiation.cc:407] Connection header received
0619 13:12:53.138110 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.138116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.138203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.138326 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.139216 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.139905 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.141107 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.141659 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.141805 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.144835 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.144840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.144843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.144878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.148077 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.148744 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.148751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.148754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.148902 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.149297 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.149303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.149306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.149658 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.149807 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.150180 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.150408 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 13:12:53.151906 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:12:53.154611 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.138743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.0.97.62:40763)
0619 13:12:53.138931 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.138937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.138956 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:12:53.139141 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.139147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.139231 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.139353 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.140604 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.141538 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.143184 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.143661 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.143821 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.148758 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.148763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.148767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.148802 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.151774 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.152530 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.152536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.152541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.152717 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.153186 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.153190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.153194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.153589 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.153902 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.154158 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.154332 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:12:53.166479 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.152636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33840)
0619 13:12:53.152800 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.152817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:53.152910 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:53.153193 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:53.153197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:53.153723 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:53.153910 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.153923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.155211 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.155216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.156184 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.156199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.157104 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.157109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.157367 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.157966 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:53.157983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:53.159957 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:53.163063 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.163069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.163073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.163553 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.164767 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.164771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.164773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.164913 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.165555 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:53.165561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:53.165789 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 13:12:53.165944 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:12:53.166183 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":23}
I20260619 13:12:53.166797 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.152701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.0.97.60:39647)
0619 13:12:53.152889 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.152895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.152913 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:12:53.152994 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.152999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.153070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.153165 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.154064 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.155045 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.156339 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.156956 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.157095 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.160145 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.160151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.160154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.160181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.162912 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.163689 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.163693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.163695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.163823 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.165025 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.165030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.165033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.165422 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.165562 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.166089 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.166362 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":81}
I20260619 13:12:53.171572 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.156288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.0.97.61:33301)
0619 13:12:53.156477 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.156483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.156498 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:12:53.156546 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.156551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.156625 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.156727 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.158024 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.158768 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.160042 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.160478 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.160614 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.164835 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.164840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.164843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.164878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.168661 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.169400 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.169406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.169410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.169565 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.169983 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.169988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.169991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.170334 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.170626 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.170811 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.171044 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:12:53.172744 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: Bootstrap starting.
I20260619 13:12:53.176602 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:12:53.176786 23398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:53.177996 23398 log.cc:824] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:53.180955 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: No bootstrap required, opened a new log
I20260619 13:12:53.179386 23225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "23c87ca8caee4944b751f9e6698cf985"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44280
I20260619 13:12:53.188377 23398 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.188727 23398 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12becb655ab43e593f5b8c2f9c7ad01, State: Initialized, Role: FOLLOWER
I20260619 13:12:53.189227 23398 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.190253 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.190644 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.191310 23398 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:53.193759 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 4
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:12:53.201332 23331 master_runner.cc:186] Error getting master registration for 127.0.97.62:40763: OK, instance_id { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" instance_seqno: 1781874772049671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:12:53.213624 23331 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:12:53.214102 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.194491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33854)
0619 13:12:53.194920 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.194943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:53.195091 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:53.196064 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:53.196069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:53.196674 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:53.196908 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.196921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.198349 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.198358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.199881 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.199902 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.201006 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.201014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.201288 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.202087 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:53.202121 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:53.205938 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:53.210914 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.210920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.210924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.211480 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.211982 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.211987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.211992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.212204 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.213116 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:53.213120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:53.213301 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 13:12:53.213465 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 13:12:53.213816 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260619 13:12:53.214800 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.195454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.0.97.60:39647)
0619 13:12:53.195654 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.195662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.195688 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:12:53.195758 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.195765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.195868 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.195984 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.197142 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.198182 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.200057 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.200592 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.200696 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.206251 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.206258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.206266 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.206321 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.210690 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.211638 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.211644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.211647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.211821 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.212342 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.212347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.212351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.212864 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.213048 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.214383 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.214557 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 13:12:53.216295 23415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:12:53.216413 23415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:12:53.220834 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: Bootstrap starting.
I20260619 13:12:53.225198 23250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:53.226646 23250 log.cc:824] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:53.230305 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: No bootstrap required, opened a new log
I20260619 13:12:53.238610 23250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.239106 23250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c87ca8caee4944b751f9e6698cf985, State: Initialized, Role: FOLLOWER
I20260619 13:12:53.239672 23250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.240806 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.241209 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.242208 23250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:53.242272 23225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:12:53.242478 23225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:53.246863 23225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.249764 23373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23c87ca8caee4944b751f9e6698cf985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d12becb655ab43e593f5b8c2f9c7ad01"
I20260619 13:12:53.250308 23373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:53.254058 23373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c87ca8caee4944b751f9e6698cf985 in term 1.
I20260619 13:12:53.254792 23199 leader_election.cc:302] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c87ca8caee4944b751f9e6698cf985, d12becb655ab43e593f5b8c2f9c7ad01; no voters:
I20260619 13:12:53.255339 23418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:12:53.261380 23225 leader_election.cc:288] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [CANDIDATE]: Term 1 election: Requested vote from peers ec6dd3a1a8a8450ba42c3bbd9bdf9d77 (127.0.97.61:33301), d12becb655ab43e593f5b8c2f9c7ad01 (127.0.97.60:39647)
W20260619 13:12:53.264781 23429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:12:53.265035 23429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:12:53.253834 23183 master_runner.cc:186] Error getting master registration for 127.0.97.61:33301: OK, instance_id { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" instance_seqno: 1781874772572195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:12:53.270987 23183 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:12:53.274015 23418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [term 1 LEADER]: Becoming Leader. State: Replica: 23c87ca8caee4944b751f9e6698cf985, State: Running, Role: LEADER
I20260619 13:12:53.274674 23418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.282210 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c87ca8caee4944b751f9e6698cf985. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.282864 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: This master's current role is: LEADER
W20260619 13:12:53.288852 23204 tablet.cc:2369] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:53.296074 23435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:12:53.301622 23435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:12:53.303862 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } attempt: 5
Jun 19 13:12:53 dist-test-slave-q11v krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:12:53.324712 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.305297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.0.97.60:39647)
0619 13:12:53.305474 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.305480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.305499 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:12:53.305566 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.305576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.305678 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.305921 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.307526 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.308733 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.310844 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.311553 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.311682 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.316146 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.316154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.316159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.316202 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.320094 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.321034 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.321040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.321043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.321200 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.321670 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.321675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.321679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.322029 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.322149 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.322896 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.323156 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:12:53.324221 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.304519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39647 (local address 127.0.0.1:33870)
0619 13:12:53.304927 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 13:12:53.304948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:12:53.305073 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:12:53.306528 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:12:53.306533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:12:53.307097 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:12:53.307321 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.307336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.308910 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.308916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.310607 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:12:53.310625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.311909 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.311916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:12:53.312120 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.312885 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:12:53.312911 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:12:53.315913 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:12:53.320281 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.320289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.320293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.320872 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.321340 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:12:53.321345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:12:53.321349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:12:53.321523 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:12:53.322253 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:12:53.322260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:12:53.322478 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 13:12:53.322630 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:12:53.323774 (+ 1144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260619 13:12:53.327834 23373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [term 1 FOLLOWER]: Refusing update from remote peer 23c87ca8caee4944b751f9e6698cf985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:53.328925 23431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:53.331068 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: Bootstrap starting.
W20260619 13:12:53.333189 23352 tablet.cc:2369] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:12:53.335297 23324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: Neither blocks nor log segments found. Creating new log.
I20260619 13:12:53.336655 23324 log.cc:824] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: Log is configured to *not* fsync() on all Append() calls
I20260619 13:12:53.352523 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.299401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.0.97.62:40763)
0619 13:12:53.299584 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.299590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.320991 (+ 21401us) server_negotiation.cc:407] Connection header received
0619 13:12:53.321494 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.321499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.321588 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.321682 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.324467 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.325518 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.339308 (+ 13790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.340283 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.340440 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.344368 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.344377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.344389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.344445 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.348404 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.349477 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.349486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.349493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.349702 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.350275 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.350282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.350288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.350719 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.350997 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.351665 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.351986 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:12:53.357384 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: No bootstrap required, opened a new log
I20260619 13:12:53.381233 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c87ca8caee4944b751f9e6698cf985. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.381636 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.384219 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.328814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.0.97.60:39647)
0619 13:12:53.337395 (+ 8581us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.337401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.337423 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:12:53.337491 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.337497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.337638 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.337763 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.340169 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.341381 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.345687 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.346380 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.346519 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.351354 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.351361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.351366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.351407 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.369066 (+ 17659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.371178 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.371189 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.371194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.372019 (+ 825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.373096 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.373103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.373106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.374843 (+ 1737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.377113 (+ 2270us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.377800 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.379258 (+ 1458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8457,"thread_start_us":12703,"threads_started":1}
I20260619 13:12:53.384511 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.306265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.0.97.61:33301)
0619 13:12:53.306436 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.306441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.324253 (+ 17812us) server_negotiation.cc:407] Connection header received
0619 13:12:53.324411 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.324416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.324491 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.324585 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.353923 (+ 29338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.355051 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.366015 (+ 10964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.366862 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.366988 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.375289 (+ 8301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:12:53.375298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:12:53.375303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:12:53.375357 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:12:53.379260 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.380171 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.380178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.380184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.380369 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:12:53.381809 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:12:53.381815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:12:53.381820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:12:53.382276 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:12:53.382517 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.383394 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.383838 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:12:53.387153 23440 mvcc.cc:205] Tried to move back new op lower bound from 7298559071535341568 to 7298559071338827776. Current Snapshot: MvccSnapshot[applied={T|T < 7298559071535341568}]
I20260619 13:12:53.388763 23324 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.389182 23324 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6dd3a1a8a8450ba42c3bbd9bdf9d77, State: Initialized, Role: FOLLOWER
I20260619 13:12:53.389779 23324 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } }
I20260619 13:12:53.402201 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.402952 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.403744 23324 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:12:53.403774 23299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23c87ca8caee4944b751f9e6698cf985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77"
I20260619 13:12:53.404285 23299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:12:53.408499 23299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c87ca8caee4944b751f9e6698cf985 in term 1.
I20260619 13:12:53.410071 23298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [term 1 FOLLOWER]: Refusing update from remote peer 23c87ca8caee4944b751f9e6698cf985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:12:53.412154 23418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:12:53.417505 23435 catalog_manager.cc:1347] Generated new cluster ID: e7698916057748229b89460c08f32ef3
I20260619 13:12:53.417640 23435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:12:53.432351 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.432700 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:53.451167 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.451526 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.458292 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c87ca8caee4944b751f9e6698cf985. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.458782 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.458320 23454 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.459118 23454 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:12:53.486956 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23c87ca8caee4944b751f9e6698cf985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c87ca8caee4944b751f9e6698cf985" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40763 } } peers { permanent_uuid: "ec6dd3a1a8a8450ba42c3bbd9bdf9d77" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33301 } } peers { permanent_uuid: "d12becb655ab43e593f5b8c2f9c7ad01" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39647 } } }
I20260619 13:12:53.488808 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985 [sys.catalog]: This master's current role is: LEADER
I20260619 13:12:53.488832 23435 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:12:53.491607 23435 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:12:53.509281 23465 catalog_manager.cc:1259] Loaded cluster ID: e7698916057748229b89460c08f32ef3
I20260619 13:12:53.509403 23465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: loading cluster ID for follower catalog manager: success
I20260619 13:12:53.513834 23435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c87ca8caee4944b751f9e6698cf985: Generated new TSK 0
I20260619 13:12:53.514151 23465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: acquiring CA information for follower catalog manager: success
I20260619 13:12:53.515067 23435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:12:53.516618 23465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec6dd3a1a8a8450ba42c3bbd9bdf9d77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:12:53.654054 23254 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.630468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.0.97.62:40763)
0619 13:12:53.630737 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.630747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.651492 (+ 20745us) server_negotiation.cc:407] Connection header received
0619 13:12:53.652325 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.652333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.652430 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.652511 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.653785 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44310: BlockingRecv error: recv got EOF from 127.0.0.1:44310 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 13:12:53.656805 23328 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.636562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.0.97.61:33301)
0619 13:12:53.636769 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.636776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.654840 (+ 18064us) server_negotiation.cc:407] Connection header received
0619 13:12:53.655137 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.655144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.655300 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.655387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.656530 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54980: BlockingRecv error: recv got EOF from 127.0.0.1:54980 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 13:12:53.660403 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.658295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.0.97.60:39647)
0619 13:12:53.658540 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.658547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.658950 (+ 403us) server_negotiation.cc:407] Connection header received
0619 13:12:53.659121 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.659127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.659230 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.659346 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.660136 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33896: BlockingRecv error: recv got EOF from 127.0.0.1:33896 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:12:53.709254 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.700197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.0.97.61:33301)
0619 13:12:53.700462 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.700472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.700990 (+ 518us) server_negotiation.cc:407] Connection header received
0619 13:12:53.701392 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.701398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.701717 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.701876 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.702624 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.703772 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.705936 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.706647 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.706749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.707573 (+ 824us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:53.708358 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:53.708481 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.708776 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.708887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 13:12:53.769387 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.754313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.0.97.60:39647)
0619 13:12:53.754577 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.754587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.767244 (+ 12657us) server_negotiation.cc:407] Connection header received
0619 13:12:53.767556 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.767562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.767711 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.767821 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.769092 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33912: BlockingRecv error: recv got EOF from 127.0.0.1:33912 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:12:53.817663 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.802825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.0.97.61:33301)
0619 13:12:53.803039 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.803047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.810496 (+ 7449us) server_negotiation.cc:407] Connection header received
0619 13:12:53.811057 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.811065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.811198 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.811288 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.812014 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.813274 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.815396 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.816077 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.816174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.816817 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:53.816961 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:53.817022 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.817235 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.817323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 13:12:53.871455 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.858386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.97.60:39647)
0619 13:12:53.858589 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.858596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.869404 (+ 10808us) server_negotiation.cc:407] Connection header received
0619 13:12:53.869978 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.869985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.870084 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.870155 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.871208 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33920: BlockingRecv error: recv got EOF from 127.0.0.1:33920 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:12:53.902595 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.896790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.0.97.61:33301)
0619 13:12:53.897055 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.897068 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.897658 (+ 590us) server_negotiation.cc:407] Connection header received
0619 13:12:53.897917 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.897923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.898015 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.898129 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.898587 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.899384 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.900744 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.901190 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.901334 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.901763 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:53.901880 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:53.901989 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.902144 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.902242 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 13:12:53.920046 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.909367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.97.60:39647)
0619 13:12:53.909626 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.909635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.918224 (+ 8589us) server_negotiation.cc:407] Connection header received
0619 13:12:53.918478 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.918484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.918594 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.918740 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.919769 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33930: BlockingRecv error: recv got EOF from 127.0.0.1:33930 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:12:53.934257 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.927649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.0.97.61:33301)
0619 13:12:53.927898 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.927909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.927985 (+ 76us) server_negotiation.cc:407] Connection header received
0619 13:12:53.928274 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.928281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.928382 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.928489 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.928957 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.930145 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.932127 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.932837 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.932995 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.933459 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:53.933615 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:53.933712 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.933868 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.933962 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260619 13:12:53.946539 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.943969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.0.97.60:39647)
0619 13:12:53.944247 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.944253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.944826 (+ 573us) server_negotiation.cc:407] Connection header received
0619 13:12:53.945063 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.945069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.945186 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.945303 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.946285 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33932: BlockingRecv error: recv got EOF from 127.0.0.1:33932 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
I20260619 13:12:53.967805 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.960638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.97.61:33301)
0619 13:12:53.960818 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.960825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.961367 (+ 542us) server_negotiation.cc:407] Connection header received
0619 13:12:53.961635 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.961643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.961733 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.961823 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.962314 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.963425 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.965455 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.966135 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:53.966259 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:53.966791 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:53.966977 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:53.967076 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:53.967292 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 13:12:53.967376 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 13:12:53.976809 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:53.974882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.0.97.60:39647)
0619 13:12:53.975080 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.975086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.975308 (+ 222us) server_negotiation.cc:407] Connection header received
0619 13:12:53.975490 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.975493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.975564 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.975646 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:53.976548 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33936: BlockingRecv error: recv got EOF from 127.0.0.1:33936 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:12:54.002825 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:53.996674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.97.61:33301)
0619 13:12:53.996904 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:53.996910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:53.997272 (+ 362us) server_negotiation.cc:407] Connection header received
0619 13:12:53.997443 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:53.997450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:53.997560 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:53.997716 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:53.998125 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:53.999325 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.000655 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.001386 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.001528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.002033 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.002173 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.002268 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.002407 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.002508 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 13:12:54.011790 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:54.009242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.97.60:39647)
0619 13:12:54.009522 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.009532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.010107 (+ 575us) server_negotiation.cc:407] Connection header received
0619 13:12:54.010405 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.010411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.010521 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.010661 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:54.011528 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33952: BlockingRecv error: recv got EOF from 127.0.0.1:33952 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 13:12:54.054790 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.047981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.97.61:33301)
0619 13:12:54.048221 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.048226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.048581 (+ 355us) server_negotiation.cc:407] Connection header received
0619 13:12:54.048875 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.048880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.048966 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.049061 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.049585 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.050542 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.052608 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.053154 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.053271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.053892 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.054023 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.054094 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.054324 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.054402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 13:12:54.067958 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:54.064038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.97.60:39647)
0619 13:12:54.064317 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.064327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.066209 (+ 1882us) server_negotiation.cc:407] Connection header received
0619 13:12:54.066521 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.066525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.066677 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.066791 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:54.067752 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33968: BlockingRecv error: recv got EOF from 127.0.0.1:33968 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 13:12:54.140762 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.135823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.97.61:33301)
0619 13:12:54.136131 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.136138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.136372 (+ 234us) server_negotiation.cc:407] Connection header received
0619 13:12:54.136543 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.136548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.136620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.136709 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.137076 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.137874 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.139182 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.139608 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.139705 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.140064 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.140160 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.140244 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.140371 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.140436 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
W20260619 13:12:54.150661 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:12:54.147842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.97.60:39647)
0619 13:12:54.148160 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.148169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.148991 (+ 822us) server_negotiation.cc:407] Connection header received
0619 13:12:54.149284 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.149290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.149384 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.149501 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:12:54.150414 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33980: BlockingRecv error: recv got EOF from 127.0.0.1:33980 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 13:12:54.218608 23415 catalog_manager.cc:1259] Loaded cluster ID: e7698916057748229b89460c08f32ef3
I20260619 13:12:54.218756 23415 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: loading cluster ID for follower catalog manager: success
I20260619 13:12:54.221383 23415 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: acquiring CA information for follower catalog manager: success
I20260619 13:12:54.223477 23415 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d12becb655ab43e593f5b8c2f9c7ad01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:12:54.288792 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.283248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.97.61:33301)
0619 13:12:54.283518 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.283525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.283770 (+ 245us) server_negotiation.cc:407] Connection header received
0619 13:12:54.283885 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.283889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.283963 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.284049 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.284494 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.285257 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.286690 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.287221 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.287434 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.287860 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.287977 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.288052 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.288405 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.288517 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 13:12:54.301262 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.295210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.0.97.60:39647)
0619 13:12:54.295531 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.295541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.295857 (+ 316us) server_negotiation.cc:407] Connection header received
0619 13:12:54.296027 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.296032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.296187 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.296306 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.296697 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.297455 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.298851 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.299317 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.299436 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.300005 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.300579 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.300701 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.300841 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.300916 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 13:12:54.316094 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.307539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.0.97.61:33301)
0619 13:12:54.307863 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.307873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.308550 (+ 677us) server_negotiation.cc:407] Connection header received
0619 13:12:54.308711 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.308725 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.308846 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.308926 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.310533 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.311680 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.314013 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.314688 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.314818 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.315287 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.315434 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.315531 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.315694 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.315798 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:12:54.317420 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.308630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.0.97.60:39647)
0619 13:12:54.308842 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.308848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.310728 (+ 1880us) server_negotiation.cc:407] Connection header received
0619 13:12:54.310921 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.310926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.311011 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.311103 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.311675 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.312871 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.315054 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.315727 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.315841 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.316719 (+ 878us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.316824 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.316930 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.317109 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.317186 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:12:54.317476 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:12:54.307306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.0.97.62:40763)
0619 13:12:54.308014 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 13:12:54.308022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:12:54.308676 (+ 654us) server_negotiation.cc:407] Connection header received
0619 13:12:54.308826 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:12:54.308845 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:12:54.309091 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:12:54.309280 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:12:54.310369 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.311537 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.314979 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:12:54.315534 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:12:54.315664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:12:54.316336 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:12:54.316787 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:12:54.316904 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:12:54.317059 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:12:54.317153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":449,"threads_started":1}
I20260619 13:12:54.322486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 23183
I20260619 13:12:54.324425 23245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:54.439704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 23183
W20260619 13:12:54.460758 23276 connection.cc:535] server connection from 127.0.0.1:54938 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 13:12:54.460800 23348 connection.cc:535] server connection from 127.0.0.1:33854 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:12:54.461570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 23257
I20260619 13:12:54.462786 23319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:54.570693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 23257
I20260619 13:12:54.591223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu pid 23331
I20260619 13:12:54.592430 23393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:12:54.700237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwPkIj_/build/asan/bin/kudu with pid 23331
2026-06-19T13:12:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4522 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9506 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369320 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST